1. Gary Langer
Langer Research Associates
glanger@langerresearch.com
NYAAPOR 6/6/13

2. Fasten your…

3. The Seat Belt (courtesy of Floyd Fowler)
When you sit in the back seat of a car, how often do you
use the seat belt – always, sometimes, rarely or never?
The problem:
A quarter of Americans never sit in the back seat of a car.

4. Their options:
a) Tell you, uninvited, that they never sit in the back seat.
b) Think of what they usually do in the front seat, figure
they’d also do that if they ever sat in the back seat (which
they don’t), and tell you they guess they’d do that.
c) Use the offered option that best approaches reality:
“Never.” But meaning that they never sit in the back seat,
not that they never use their seat belt when they do.
Because they don’t.
What they do: c)

5. The result:
Vast overstatement of the non-use of back-seat seatbelts.

6. What this is about:
a) Presumption
b) Cognition
Presumption, meaning:
 We know what we want to find out, so, dammit, why
doesn’t everyone else?
Cognition, meaning:
 How people think about, understand and answer our
questions.
 And how we make sense of their answers.

8.  Have you ever seen anything that you thought was a UFO, or
not? Gallup 6/90
Saw UFO: 14%
 Have you ever seen anything that you thought was a UFO, or
not? ABC 6/97
Saw UFO: 13%
 (If yes:) Do you believe what you saw was a spacecraft from
another planet, or do you think it was something man-made
or some natural phenomenon that you could not identify?
Thought it was an alien spacecraft: 4%

9. The risks of non-specification
 We can project meaning where it’s unintended by the
respondent.
 The respondent can project meaning where it’s
misinterpreted by us.

10. Expressed (vs. affirmed) belief
 Before the war, do you think Iraq did or did not
provide direct support to the Al Qaeda terrorist
group? ABC/Post ’05
Did: 61%
 If yes: (Is that your suspicion only), or (do you
think there's been solid evidence of that)?
Evidence: 21%

11. Is President Obama
… A Muslim?
… Kenyan-born?
…
“Poll: 1 in 5 Americans Believe Obama is a Cactus”
(The Onion, 2010)
(See also climate change.)

12. Context, too
 Trust in government
…to do what?
 Pre 9/11: Muck around with social policy
 Post 9/11: Keep us from getting killed by terrorists
 Post 2008: Screw up the economy

13. The solution
Respect this thing.

14. The best way to stay out of trouble
Don’t ask foolish questions.
(Yes. In polling, there are foolish questions.)

15. How we (try to) avoid them
• Learn from the literature
• Acquiescence bias
• Agree/disagree, yes/no, true/false
• Satisficing/cognitive burden
• Positive attribute bias
• Optimal vs. suboptimal response scales
• Numbers/words, categories, labeling, branching
 The midpoint controversy (resolution may be purpose-driven)
• Priming
• Order effects/list effects
• Good news soon, at a bookstore near you

16.  Mind the basics
 Balance
 Proportion
 Cover the options
 Open-ends can help find them
 Mutual exclusivity (e.g. walk/lunch)
 Alt. prop.
 Simplicity
 Clarity
 Brevity
 Intros can stack the cards. Some/other people can, too
 Avoid jargon, euphemisms and hot buttons
 Be aware of kvetchism and the danger of retroactive trend
 Know the limits of recall
Further foolishness avoidance

17. Know thy purpose and find a path
 Practice risk assessment:
 Where will this go wrong?
 Know who you’re talking to
 Know what you’re trying to find out
 Research what’s known, but skeptically
 Experiment
 Stack not the cards, load not the dice
 Recognize the limitations of any single question

18. Have a seat…

19.  Do you support or oppose the death penalty for
convicted murderers? (68%)
 What do you prefer for convicted murderers, the
(death penalty) or (life in prison with no possibility
of parole)? (49%)
 Do you support or oppose the death penalty for…
…Timothy McVeigh (75%)
…Murderers who are mentally retarded? (19%)
…Murderers who were juveniles at the time of the
crime? (26/49/23%) (Note the vols!)

20. Same with abortion… and so many others….
 Support or oppose legal abortion? (55%)
…To protect woman’s life? (88%)
…In cases of rape or incest (81%)
…To terminate unwanted pregnancy? (42%)
…After 1st trimester (11%)
…Private matter? Murder?

21. Dare disagree?
 Do you agree or disagree with this statement: “Abortion is murder.”
LA Times 3/89
Agree: 58%
 Which of these statements comes closer to your opinion –abortion
is the same thing as murdering a child, or abortion is not murder
because the fetus really isn't a child? CBS/NYT 4/89
Murder: 48%
 Some people say that abortion is an act of murder, while other
people disagree with this. What is your view – do you think
abortion is an act of murder or don't you feel this way? (Time/CNN 8/94)
Murder: 42%

22. Thanks, Willem and Jon
“A huge body of research conducted during more than five
decades has documented the role that acquiescence response
bias plays in distorting answers to agree/disagree questions.”
“… remarkably sizable differences in data quality” in
agree/disagree vs. balanced forced-choice formats.
-Saris and Krosnick, “Comparing Questions with Agree/Disagree Response
Options to Questions with Construct-Specific Response Options.”

23. Why?
 Absence of the alternative proposition is inherently
unbalanced;
 Common “solution,” rotating positive and negative
statements, merely shifts the bias rather than resolving it;
 Format encourages acquiescence; and
 Satisficing to minimize cognitive burden.

25. Why? For a headline, maybe
“Significant numbers of African Americans believe in
conspiracy theories about AIDS…” RAND Corp./Oregon State
University, 1/05
“More than 29 years after the AIDS virus arrived in the
United States, a significant proportion of African
Americans embrace the theory that government
scientists created the disease to control or wipe out
their communities, according to a study released today
by Rand Corp. and Oregon State University.” Washington Post,
1/25/05

26. First, I’m going to read you several statements regarding HIV and AIDS. For
each one, I’d like you to tell me whether you agree strongly, agree somewhat,
have no opinion, disagree somewhat, or disagree strongly. The first one is...
1.1 The medicines used to treat HIV are saving lives in the black community.
1.2 A lot of information about AIDS is being held back from the public.
1.3 HIV is a manmade virus.
1.4 There is a cure for AIDS, but it is being withheld from the poor.
1.5 The government is telling the truth about AIDS.
1.6 The medicine used to treat HIV causes people to get AIDS.
1.7 HIV was created and spread by the CIA.
1.8 AIDS is a form of genocide against Blacks.
1.9 The medicine that doctors prescribe to treat HIV is poison.
1.10 AIDS was created by the government to control the Black population.
1.11 Doctors put HIV into condoms.
1.12 People who take the new medicines for HIV are human guinea pigs for the government.
1.13 Medical and public health institutions are trying to stop the spread of HIV in
black communities.
1.14 AIDS was produced in a government laboratory.

27. Further headaches…
 “Significant proportion” was 16 percent in agree/disagree.
 Sample, described as “a random sample of 500 African-
Americans,”* was limited to high-density (>26%) black
Census tracts; noncoverage 49.5%.
 Survey introduction twice says it’s “about discrimination”
 Sample limited to 15- to 44-year-olds.
*Journal of Acquired Immune Deficiency Syndrome, 2/05

28.  With the economy the way it is, some women may have
changed their plans about when or how many children to
have. Is the following statement true for you? Because of
the economy, I now want to have a baby later than I had
previously planned Guttmacher Institute, 9/09 (Sexually active women 18-34 <$75K)
True: 44%
 As a result of the economic slump, have you postponed or
canceled any of the following? Having a baby Time 4/09, women
18-34 <$75K (N=56)
Yes: 18%
Y/N, T/F, too

29. Priming the pump…
 As a result of the recent increase in gas prices have you...cut
back on your driving in any way? Newsweek 9/05
Driving less: 71%
 Thinking about how much driving you do, would you say
that compared to this time last year you're driving more,
less, or about the same amount? (If less, ask:) Is that
because of the price of gasoline, or for some other reason?
ABC/Post 8/05
Driving less: 31% (Because of gas prices, 22%)

30. Did I say aagh?
Stephen,
We recently used a very similar question to that used by Paul for a dual-
frame survey in Australia …
“Research companies sometimes invite members of the general public to
become members of online research panels where they receive points or
incentives for regularly completing surveys online. Are you currently a
member of a research panel?”
Cheers,
Darren
Managing Director and Head of Research…
(AAPORNET, 6/5/13)

31. Let’s try
Research companies sometimes invite members of
the general public to become members of online
research panels where they receive points or
incentives for regularly completing surveys online.
Are you currently a member of a research panel?
How about… (just a starter)
Do you ever take surveys over the internet in exchange
for cash or gifts, or not?

32. Let’s give this a go

33. Gay marriage
 Do you support or oppose gay marriage?
 Do you think it should be legal or illegal for gays and
lesbians to get married?
 Do you favor or oppose allowing gays and lesbians to
marry legally?
 Consider further: homosexuals, gays and lesbians,
same-sex couples, men/men and women/women?

34. The hall of horrors

35. What’s the dope?
 “As you may know, methamphetamine use has risen
dramatically in counties across the nation…
…Have arrests where methamphetamine was involved
increased in your county?”
...Which of the following drugs is the biggest problem
in your county…?
National Association of Counties, 6/05 (n.b., convenience sample)

36. “Knowledge” Polls
 Knowledge vs. recall
 Opinion vs. behavior vs. recall
 The operation and activation of memory
 The weight of error
 The effect of mode/method (No. 2 pencils)
 The attempt to humiliate

37. Just say no.
13. Socrates, Plato, Aristotle, and Aquinas would concur that:
a. all moral and political truth is relative to one’s time and place
b. moral ideas are best explained as material accidents or byproducts of evolution
c. values originating in one’s conscience cannot be judged by others
d. Christianity is the only true religion and should rule the state
e. certain permanent moral and political truths are accessible to human reason
27. Free markets typically secure more economic prosperity than government’s centralized planning
because:
a. the price system utilizes more local knowledge of means and ends
b. markets rely upon coercion, whereas government relies upon voluntary compliance with the law
c. more tax revenue can be generated from free enterprise
d. property rights and contracts are best enforced by the market system
e. government planners are too cautious in spending taxpayers’ money
29. A flood-control levee (or national defense) is considered a public good because:
a. citizens value it as much as bread and medicine
b. a resident can benefit from it without directly paying for it
c. government construction contracts increase employment
d. insurance companies cannot afford to replace all houses after a flood
e. government pays for its construction, not citizens
Intercollegiate Studies Institute 5/08

38. The Black Art:
Subtle and unsubtle manipulations

39. Sounds good…
 “A recent survey shows that two-thirds of women would gladly reduce
the number of monthly periods if they could.” (ABC PrimeTime 9/19/03)
 “If there was a safe and easy way, how interested would you be in
reducing your period and related symptoms, such as PMS or bloating,
to only four times a year or once every three months? Would you be
extremely interested, very interested, somewhat interested, not very
interested, or not at all interested in reducing your period to only four
times a year?”
• Positive bias: “safe,” “easy,” “reduced symptoms.”
• No mention of medication, cost, side effects.
• 3-2 unbalanced
• Produced in Barr Labs promotional campaign

40. Drive it away…
 Do you believe that the government should or should not provide loans
to America’s automakers so they have the money to manufacture
vehicles?
Should Should not No opin.
55% 30 15
 The big three automakers in the United States have asked for a 25
billion dollar loan from the government. Some people say (it's a bailout
those companies don't deserve, and that they'd be better off
reorganizing under bankruptcy laws). Other people say (it's necessary
to protect auto workers and save a key part of the U.S. economy). On
balance, do you support or oppose this plan?
Support Oppose No opin.
35% 57 9

41. Behind the curtain…
1. How important do you feel the American automobile industry is to the
American economy--extremely important, very important, somewhat
important, not important, or not at all important?
2. If the American automobile industry no longer had the resources to
produce vehicles, how much harm would it cause to... INSERT – a great deal
of harm, quite a bit of harm, just some harm, or very little harm?
a. America's manufacturing job sector
b. the American economy
c. America's standing in the world
d. consumer choice for America's car buyers
3. Do you believe that the government should or should not provide loans to
America's automakers so they have the money to manufacture vehicles?
Hart Research for GM, 11/08

42. Let’s Dance. I’ll lead…
 Let me read you four facts about the auto industry's effect on the American
economy.
(1) The U.S. auto industry affects numerous other American industries
because it is the largest purchaser of steel, copper, plastics, electronics, and
computer chips.
(2) The U.S. auto industry provides pensions to seven hundred and seventy-
five thousand Americans and health benefits to two million Americans.
(3) The U.S. auto industry supports approximately five million U.S. jobs in all
fifty states.
The U.S. auto industry has fourteen thousand dealers in nearly every town in
America employing seven hundred thirty thousand people.
Knowing this, if General Motors, Ford, and Chrysler were to go out of
business in the next few months, how likely do you think this would be to
trigger an economic depression in the U.S. – extremely likely, very likely,
somewhat likely, not likely, or not at all likely? (3-2 = 79%)

43. Lowdown on the down-low
Public release date: 18-Sep-2006
American College of Physicians
In new survey, men call themselves straight but have sex with men
PHILADELPHIA, September 19, 2006 – A survey of 4,193 men living in New York City conducted by the New York City
Department of Health and Mental Hygiene found that nearly 10 percent of male participants who identified themselves as
straight reported having sex with at least one man during the previous year.
The study, "Discordance between Sexual Behavior and Self-Reported Sexual Identity: A Population-Based Survey of New
York City Men," is published in the Sept. 19, 2006, issue of Annals of Internal Medicine.
Compared to men who identified themselves as gay, these men were more likely to belong to a minority racial or ethnic
group, be foreign-born, have a lower educational level, and live outside Manhattan. Seventy percent reported being
married. This group also was less likely to have been tested for HIV infection during the previous year and less likely to have
used a condom during the last sexual encounter than men who identified themselves as gay.
"Doctors need to ask patients about specific sexual practices instead of relying on self-reported sexual orientation to assess
risk for unsafe sexual practices and risk for sexually transmitted diseases," said Preeti Pathela, DrPH, lead author of the
study. "Public health prevention messages should target risky sexual activities, such as unprotected receptive anal sex, and
should not be framed to appeal solely to gay-identified men."
This study is one of the largest U.S. population-based surveys to report on the contrast between a man's self-identified
sexual identity and his actual sexual behaviors.
###
NOTE TO EDITORS: To interview Preeti Pathela, DrPH or Harold Sox, MD, Editor of Annals of Internal Medicine, contact
Steve Majewski at 215-351-2514 or smajewski@acponline.org.
43

44. 44
Limitations and Findings
 First hurdle: Sampling methodology is top drawer; study is part
of federal BRFSS.
 “Limitations: The survey did not sample groups that cannot be
reached by using residential telephone services.”
 “Conclusions: Many New York City men who have sex with men
do not identify as gay.”

45. Almost one in 10 straight men
on the “down-low,” study finds
9/18/2006
By Dawn Fallik
The Philadelphia Inquirer
Almost 10 percent of men who say they're straight also
happen to be having sex with men, according to a new study,
one of the largest ever to specifically address "down-low"
behavior.
The study, based out of New York City, found that most of the
down-low men did not use condoms and that 70 percent
were married. Researchers said they hoped their report
would change the way doctors asked patients about their
sexual behavior.
45

46. 46
What they asked
(Figure 1)
“Question: ‘During the past 12 months, with how many men have
you had sex?’
“Interviewer read the following: ‘By sex we mean oral, vaginal, or
anal sex, but not masturbation.’

47. 47
Whoa.
 Last thing respondent heard was not about sex
with men, but about sex including vaginal sex.
 Self-identified heterosexual men who reported
having sex (including vaginal sex) with other men
were disproportionately foreign born, low-
education, married (70%), apt to report a single sex
partner in the previous year (96%) and less likely to
use a condom or be HIV-tested.

48. Did they misunderstand the question?
Were they reporting sex…
with their wives?
48

49. 49
In the 30th paragraph (of 35)
 “It is possible that this unique method, in which
men were asked about same-sex partners first and
the accompanying definition of sex included
vaginal intercourse, resulted in misunderstanding
of the question…”
 “Preliminary data from the 2005 CHS, which
ascertained sex of partners by asking about
opposite-sex partners first, suggest that doing so
resulted in a much lower self-reported prevalence
of men who report sex with other men.
(…unpublished data.)”

50.  Subsequent response from the California Department of
Health Services reports a statewide estimate of 0.4 percent
of straight men on the down-low, compared with the 9.4
percent in the NYC study.
 The California researchers conclude that the NYC finding
"is inconsistent with the rates reported by others, and may
in fact be an artifact of the methods used in the survey."
http://www.annals.org/cgi/eletters/145/6/416#3918
50

51. What we want
• No games
• Neutral, balanced wording, w/ alternative
proposition.
• Simple phrasings with clear meanings
• Realistic and proportionate response options
• Rotations where appropriate
• Sensible order, respecting trend
• Appropriate instructions and codings
• Mindful questionnaire design

52. Gary Langer
Langer Research Associates
glanger@langerresearch.com
NYAAPOR 6/6/13