This document provides an overview of polls and discusses several key concepts related to interpreting and reporting on polls. It notes that polling error can be expected to be around 2-2.2 percentage points on average and discusses how to identify reliable polls based on factors like the source, sample size, and methodology. It also explains important polling concepts like margins of error, confidence intervals, and how a poll of polls can provide a more accurate picture than any single poll.

1. 1

2. OUTLINE:
1. Introduction
2. Polling errors
3. How to identify good and bad polls
4. Important concepts around polls
5. Poll of polls
6. The 2019 European Parliament elections on politico.eu
7. How to report polls

3. POLLS HAVE A BAD
REPUTATION
WITH GOOD REASON?
3
INTRODUCTION

4. THE POLLS ARE AL(L)RIGHT
02-04
4

5. THE POLLING ERROR
1. Can be expected to be around 2-2.2
percentage points
2. Is very consistent over time and across
elections.
3. Can be factored in.
4. Has to be clearly communicated.
5
THE POLLS ARE AL(L)RIGHT
Source: Jennings and Wlezien (2018) https://doi.org/10.1038/s41562-018-0315-6

6. POLL ERRORS ACROSS
ELECTORAL HISTORY
a: the absolute error for all
parties/candidates.
• Hollow grey circle: error for each party
• Black circle: mean absolute error across
all parties
b: the mean absolute error.
c: the mean absolute error on the margin.
d: the mean absolute value of the log odds
ratio.
e: the variance of the mean absolute error.
f: the variance of the mean absolute error
6
THE POLLS ARE AL(L)RIGHT
Source: Jennings and Wlezien (2018) https://doi.org/10.1038/s41562-018-0315-6

7. POLLING ERROR FOR THE AUSTRIAN
PARLIAMENTARY ELECTION
• Average 1.35 percentage points
• Polls with larger sample size and shorter
field period perform better
7
THE POLLS ARE AL(L)RIGHT

8. POLLING ERROR IN
DENMARK
• 1.36 percentage points polling error
• Timing matters
• Method matters
• Quality matters
8
THE POLLS ARE AL(L)RIGHT

9. 9
BUT WHAT ABOUT TRUMP AND BREXIT?
Brexit:
• The day before the referendum poll of polls showed 48% for remain 45% for leave, but 10 undecided!
THE POLLS ARE AL(L)RIGHT
Source: https://en.wikipedia.org/wiki/Opinion_polling_for_the_United_Kingdom_European_Union_membership_referendum
A CLOSER LOOK
2016 US Presidential election:
• Chances 70:30 – trump would win every third election. Would you get on a plane if every third would crash?
• National polls were correct for popular vote, but electoral college is what matters.
• Polls in key swing states with above average polling error overall 5 percentage points absolute error on average.
• Trump won the election with very small margin in important states.

10. 10
BUT WHAT ABOUT TRUMP AND BREXIT?
Brexit:
• The weeks before the referendum poll of polls showed 48% for remain 45% for leave, but 10 undecided! All polls:
THE POLLS ARE AL(L)RIGHT
Source: https://en.wikipedia.org/wiki/Opinion_polling_for_the_United_Kingdom_European_Union_membership_referendum
A CLOSER LOOK

11. 11
HOW TO IDENTIFY GOOD AND BAD POLLS
6 KEY CRITERIA
• Source: In most cases a news outlet commissions the polls and publishes it in an article, trustworthy newspaper?
• Field work: When was the poll conducted?
• Sample size: How many people where interviewed? Rule of thumb: >800
• Methodology: Was the poll conducted online, in face-to-face interviews or via telephone?
• Number of undecided voters: Do the figures add up to 100% or is the % of undecided published?
THE POLLS ARE AL(L)RIGHT

12. “REPRESENTATIVE”
12
• Each person in the group in question has the same
change to be interviewed.
• Has nothing to do with sample size
• Has nothing to do with population size either
• How to chose representative sample?
• Two methods: random sampling or quota sampling
• E.g. randomly drawn phone numbers
• Set quotas for gender, age, etc., which have to be
met in the polling region, used in online panels.
• Weighting for underrepresented , non-response
adjustment.
A good (=representative) small sample is better than a
bad large sample!
To taste a soup you don’t have to empty the pot, a
spoonful will do!
IMPORTANT CONCEPTS WHEN REPORTING ABOUT POLLS
THE POLLS ARE AL(L)RIGHT

13. THE MARGIN OF ERROR CONFIDENCE INTERVAL
13
• tells you how many percentage points your results will
differ from the real population value.
• the range of values below and above the sample
statistic in a confidence interval.
• For simple random sample approximated:
• For the percentage value “p” of a specific party in a poll
• Therefore, smaller for smaller parties.
• a way to show what the uncertainty is with a certain
poll.
• E.g. 1000 people interviewed, party A at 33% → Margin
of error is +/- 2.9 percentage points → Confidence
interval of 30.1 and 35.9.
• if the poll is repeated using the same techniques, 95%
of the calculated intervals will include the “true”
population value.
IMPORTANT CONCEPTS WHEN REPORTING ABOUT POLLS
THE POLLS ARE AL(L)RIGHT
Note: https://www.langerresearch.com/moe/ offers a margin of error calculator
𝑀𝑜𝐸 ≅
0.98
√𝑛
𝑀𝑜𝐸 = 1.96
0.33(1 − 0.33)
1000
= 2.91% → +/- MoE → Confidence Interval
𝑀𝑜𝐸 = 1.96
𝑝(1 − 𝑝)
𝑛

14. 14
OTHER ERRORS
THE POLLS ARE AL(L)RIGHT
Other sources of potential bias:
• Sampling Error – because you don’t poll the entire population
• Coverage Error – some groups can’t be polled
• Measurement Error – bad wording of question, bad instruments etc.
• Non-Response Error – bias because of who responds to poll and who doesn’t.
• All together = Total Survey Error

15. 15
AN EXAMPLE FOR THE MEASUREMENT ERROR
THE POLLS ARE AL(L)RIGHT
Source: https://twitter.com/alemannoEU

16. WHY A POLL OF POLLS IS SUPERIOR
05
16

17. PARABLE OF THE THREE
RINGS (OR POLLSTERS)
Which poll is the one to trust?
17
WHICH POLL IS THE ONE TO TRUST?
#POLLOFPOLLS

18. IS THERE A HOUSE
EFFECT?
• Polls ahead of the Swedish parliament
election 2018 by pollster reveal significant
differences.
• YouGov and Sentio constantly
overestimated Sweden democrats.
• Their method? – Online polls only
• In 2014 all pollsters underestimated
Sweden Democrats, YouGov and Sentio to
a lesser extent
• Shy Sweden Democrats-supporters?
• Less reluctant to reveal voting intention in
“anonymous” online polls?
• In 2018 overcorrected or did response
behavior change?
18
#POLLOFPOLLS

19. COMBINING POLLS REVEALS TRENDS AND OUTLIERS
19
#POLLOFPOLLS
Source: pollofpolls.eu/FR
NOT ONLY FOR VOTING INTENTION
• Don’t cherry-pick one poll, where there
are more.
• Our aggregation model uses the so-called
Kalman filtering approach + smoothing to
track the trends in the polls.
Kalman filter can be explained with
“GPS”-analogy:
• Support for a party is the “true” location
• In your Map-app, the blue circle around
your GPS-location is the uncertainty,
every new signal (poll) from the satellite
(polling firm) improves the location and
who will find your way.

20. THE 2019 EUROPEAN PARLIAMENT ELECTIONS
06
20

21. POLITICO EUROPEAN ELECTIONS HUB
21
#EP2019
Source: politico.eu/EU2019

22. POLITICO EUROPEAN
ELECTIONS HUB
• The only daily updated seat estimate
• Interactive
• Coalition builder
• Figures with and without UK
• Filter by country
• Map view
22
#EP2019
Source: politico.eu/EU2019

23. OUR SEAT PROJECTION OVER TIME
23
#EP2019
• Relatively stable
• Clear from beginning that
EPP+S&D < 50% of seats
• Biggest shifts due to group changes of
individual parties
• Brexit analysis
• Challenge was how to deal with new
groups and (unofficially) announced
group changes.

24. EUROPEAN PARLIAMENT ELECTION RESULTS
24
#EP2019
Source: politico.eu/EU2019
28 COUNTRY PAGES

25. 25
HOW ACCURATE WERE THE POLLS?
• Overall very good
• Big upsets in Slovakia (PS+SPOLU), Lithuania
(LVZS)
• Green parties underestimated in Ireland, Austria,
Germany.
• Almost no polls in the Netherlands
• No polls in Luxembourg
• No polls after Ibizagate broke in Austria
#EP2019
POLLING ERROR ACROSS
ALL 28 EU COUNTRIES

26. EUROPEAN PARLIAMENT ELECTION RESULTS
26
#EP2019
Source: politico.eu/EU2019
LAST UPDATED 21 JUNE

27. HOW TO REPORT POLLS
07
27

28. HOW TO REPORT POLLS
1. Check for key criteria
2. Report key criteria (polling firm,
commissioned by, field period, sample
size, method)
3. Look at figures with margin of error in
mind
4. Report trend compared to last poll from
same polling firm
5. Don’t overinterpret small number
1. One percentage point lead is not a
lead
2. One percentage point change is not a
real change but statistical noise
28
#CASESTUDY

29. 29
FROM POLLS TO HEADLINES
“Italians in 2019 hate chicken, Greek salad overtaking fish!”
When looking at polling results, always look for the PQRST:
P. Polling Firm: Cornelius Hirsch? Never heard of that polling firm, also can’t find a website or another published poll.
Q. Question: Chicken was not polled, international respondents.
R. Results: Others and/or undecided are not reported
S. Sample size: <800 people polled, <400 declared voting intention
T. Timing: One day polling period in June 2019.
#CASESTUDY

30. 30
FROM POLLS TO HEADLINES
(Bloomberg) — Greece's main opposition new democracy party would win parliamentary elections with a 9 percentage
point lead if they were held now, according to a pulse RC poll for Skai TV.
New democracy, a conservative party, would get from 31.5% to 36.5% of the vote compared with 22.5% to 27.5% for the
governing leftist SYRIZA party, according to the poll. The survey is the first since prime minister Alexis Tsipras called a
snap national ballot following his party’s defeat in European elections a week ago.
The center-left movement for change would come third with 6.5% to 9.5% of the vote, the communist party of Greece
fourth with 4.5% to 6.6%, the far-right golden dawn party fifth with 4% to 6% and Greek solution, a new nationalist party,
placed sixth with 3% to 5%. Mera25, the new party of former finance minister Yanis Varoufakis would receive from 2% to
4% of the vote. Parties need to reach the threshold of 3% to elect a member to parliament.
Pulse RC surveyed 1,358 people from may 30 to June 1. The margin of error was +/- [XY] Calculate margin of error!
Key characteristcs?
#CASESTUDY
Source: https://nationalpost.com/news/world/new-democracy-wins-first-poll-since-tsiprass-snap-election-call
INVENT A TITLE

31. POLL OF POLLS FOR THE 2019 GREEK SNAP ELECTION
31
#CASESTUDY
Source: pollofpolls.eu/GR
ONE POLL CAN BE AN OUTLIER, IT’S ALWAYS BETTER TO LOOK AT THE POLL OF POLLS
Interpretation?
Significant lead? → Larger than 2 x MoE
(conservative estimate 2 percentage points)
→ Yes
Increasing trend? In Jan – May 2019 ND
around 36-37%, now 39% → “Slightly”
Golden Dawn significantly decreasing.
Electoral threshold in Greece of 4%, small
parties significantly below or above?

32. 32
RECAP
Polls are
…accurate, with an average margin of error of around 2 percentage points.
…only a snapshot in time.
…more insightful when aggregated, revealing trends and outliers.
…one tool of many to assess an election campaign.
…like a fine perfume, you should smell it, but not drink it.
Surveys on twitter or on websites, where anyone can respond as often as they like, are “clickers” – they are not “polls”.
To Remember
• Margin of Error depends on sample size and not on population size.
• Confidence Interval shows uncertainty.
• Representativeness depends on chance to get interviewed, not on sample size or population size.

33. THANK YOU FOR YOUR ATTENTION
LET’S STAY IN TOUCH
@VollCornHirsch
chirsch@politico.eu
pollofpolls@politico.eu

34. POLLING ERROR BY PARTY
34
#EP2019