Random digit dialing cell phone surveys and surveillance systems: Assessing data quality and bias

Random digit dialing cell phone surveys and
surveillance systems: data quality, data
ill t d t lit d t
weighting strategies, and bias

Cristine D. Delnevo, PhD, MPH & Daniel A. Gundersen, MA,
UMDNJ‐ School of Public Health
Randal S. ZuWallack, MS & Frederica R. Conrey, PhD
Randal S ZuWallack MS & Frederica R Conrey PhD
ICF Macro International

Presented at 137th Annual Meeting & Exposition
P t d t 137th A l M ti & E iti
Philadelphia, PA
November 7‐11, 2009

Work supported in part by the National Cancer Institute (R21CA129474 ) and a contract from the New
Jersey Department of Health and Senior Services, through funding from the Cigarette Tax

Wireless substitution (US), 2005 2008
Wireless substitution (US), 2005–2008

Source: Blumberg & Luke Wireless substitution: Early release of estimates from the National Health Interview Survey, July‐
December 2008. National Center for Health Statistics. May 2009.

Demographic differences
Demographic differences
• Gender: men more likely than
Wireless substitution by age (and time)
Wireless substitution by age (and time)
women to be wireless only
b i l l
• SES: Adults living in (30.9%) and
near poverty (23.8%) more likely
than higher income adults
h hi h i d l
(16.0%) to be wireless only
• Region: Wireless substitution
highest in South (21.3%) and
hi h t i S th (21 3%) d
Midwest (20.8%) vs. Northeast
(11.4%) or West (17.2 %)
• Race/Ethnicity: Wireless
R /Eth i it Wi l
substitution highest  among
black (21.4%) and Hispanic
(25.0%) adults vs. Non‐Hispanic
(25 0%) adults vs Non Hispanic
white adults (16.6%)

Biased health estimates?
Biased health estimates?
• Potential for biased health estimates due to sample
Potential for biased health estimates due to sample
under‐coverage remains a real, growing threat to
RDD health surveys
• Cell‐phone only also differs with respect to health
behaviors and the validity of some health estimates
based on traditional RDD surveys are increasingly
questionable

Health estimates by phone status
Health estimates by phone status
Has a landline Wireless-only
y No telephone
p
telephone

NHIS July – December 2007
5+ alcoholic drinks in 1 day 17.7 37.3 27.1
Current smoker 18.0 30.6 38.6
Uninsured 13.7
13 7 28.7
28 7 44.1
44 1
Has a usual place for care 87.5 68.0 61.8
Flu vaccination 32.7 16.6 20.9
Ever tested for HIV 34.7 47.6 45.8

Source: Blumberg SJ, Luke JV. Coverage bias in traditional telephone surveys of low‐income and young
adults. Public Opin Q. 2007;71:734–749

Cigarette smoking among young adults,
2003‐2005, NHIS & BRFSS
2003 2005 NHIS & BRFSS

Source: Delnevo, Gundersen & Hagman (2008) Declining prevalence of alcohol and smoking estimates
among young adults nationally: artifacts of sample under‐coverage? Am J of Epidemiology

New challenges: Wireless Mostly?
Telephone Status, NHIS July‐
December 2008

• The percentage of adults
living in wireless‐mostly
households has been
increasing
• Who are they?
Who are they?
(demographic & health
behaviors?)
• Will they respond to
landline surveys?


Reactions?
• Starting in 2009, the Center
for Disease Control and
f Di C l d
Prevention (CDC) is
requiring states to
incorporate cell phone
incorporate cell phone
interviews in their regular
BRFSS sample
• Yet there is no widely
y
accepted methods of
evaluating data quality or
data weighting, particularly
for state and local area
for state and local area
surveys.
• AAPOR Cell Phone Task
Force report
Force report

This special session:
This special session:
• Analysis of data quality of cell phone
Analysis of data quality of cell phone
surveys
• Demonstrate weighting procedures for
merging cell phone samples with landline
samples
• An assessment of bias in landline only
An assessment of bias in landline only
surveys, and

Assessing Data Quality of
Cell Phone Random Digit
g
Dial Surveys
Frederica R Conrey
Randy Zuwallack

Data Quality

• Q lit of Responses
Quality f R
• Quality of Sample

2008 New Jersey Adult Tobacco
Survey
• R d
Random Di it Di l L d (RR 19%) and
Digit Dial Land (RR=19%) d
Cell Phone (RR=16%)
• Short version
– 49 Questions (min=37; max=70)
– 534 Cell completes
– 468 Landline completes

Survival Analysis

• Diff
Different people get diff
t l t different surveys
t
because of skip patterns
• Survival analysis
– Measures the impact of survey mode on non-
response
– Controls for differences in survey length

Item Non-Response
Failure Rates by Phone Mode
Mean
M Median
M di Std
Cell 3.3% 2.2% 7.4%
Landline 3.4% 2.0% 5.5%

Survival predicted by p
p y phone mode:
Hazard=1.00, p>.95

Open Ended Response

Total Responses p Open
p per p
Open end reports of the
O d t f th End by Phone Mode
events of 2 recalled Responses
commercials / OE DK / OE

– 251 respondents were Cell .52 .49
asked at least one Landline .48 .55
open ended question P .56 .33

Unit Non-Response

Data
D t quality i th t
lit is threatened if
d
– Response rates are low AND
– The people who DO NOT respond are
different from those who DO.
If cell phone respondents are less likely to
respond, then there is non-response bias.

Survival Analysis

• C ll and l dli respondents may get
Cell d landline d t t
different surveys
• Response rates alone don’t tell the story
• Survival analysis tells whether cell
y
respondents are more likely to break off
g
given the same survey length
y g

Survival Model

25%
Sample Breaking Off

20%
g

15%

10% Cell
e

Landline
5%

0%
0 20 40 60 80
Survey Questions
p<.001
001

What does a difference in survey
survival mean?
• C ll respondents quit sooner th l dli
Cell d t it than landline
respondents.
• The sample under-represents cell phones
• The longer the survey, the worse the
g y
nonresponse bias
• The solution?
– Careful weighting
– Short surveys

In a population study of tobacco use
behavior…
Minimal difference
Mi i l diff Substantial difference
S b t ti l diff
between cell and between cell and
landline in response landline in response
quality rate
• No difference emerged in • Sample quality may be
item-nonresponse threatened if cell phone
• No difference emerged in surveys are too long or
richness of open end
i h f d weighted incorrectly
responses.

Weighting Cell Phone Surveys
Weighting Cell Phone Surveys

Randal Zuwallack
Frederica R Conrey
Frederica R Conrey

Thanks

• Cris Delnevo
Cris Delnevo
• Dan Gundersen
• NCI (R21CA129474), New Jersey Dept of
C ( 2 C 29 ) f
Health and Senior Services

Dual Frame

D

A. Adults in landline households with no cell phone,
A Adults in landline households with no cell phone
B. Adults in landline households with a cell phone, and
C. Adults in non‐landline households with a cell phone (cell only).

Dual Frame

D

Common designs:
Common designs:
Dual frame w/ no overlap: Landline (A+B) + Cell (C)
Dual frame w/ overlap: Landline (A+B) + Cell (B+C)

Uncommon design:
Dual frame w/ no overlap: Landline (A) + Cell (B+C)
Dual frame w/ no overlap: Landline (A) + Cell (B+C)

Weighting Challenges

• Challenge 1: How do we put the dual frames
Challenge 1: How do we put the dual frames
together?

• Challenge 2: Differential Nonresponse

Challenge 1

• How do we put the dual frames together?
How do we put the dual frames together?
• No overlap
– Estimate of cell‐only population size?
Estimate of cell only population size?
• Internal estimate
• External estimate: NHIS (Blumberg et al.)
• With overlap
– Must determine group membership
– Adjust for multiple selection probabilities
– Estimate of phone group population sizes?

Cell Survey

• “In addition to your cell phone is there at
In addition to your cell phone, is there at
least one telephone inside your home that is
currently working and is not a cell phone? Do
currently working and is not a cell phone? Do
not include telephones only used for business
or telephones only used for computers or fax
or telephones only used for computers or fax
machines.”
– ‘yes’ = dual user, while those who responded
yes = dual user while those who responded
– ‘no’ = cell‐only

Landline Survey

• “In addition to your residential landline
In addition to your residential landline
telephone, do you also use one or more cell
phone numbers?
phone numbers?”
– ‘yes’ = dual user
– ‘no’ = landline only.
no = landline only

Example 1‐‐Colorado

• Combine with BRFSS
Combine with BRFSS
• Group membership
–KKnown for cell
f ll
– Unknown for landline
• Limited to dual frame w/ no overlap
– Used 15% (NHIS state estimates) for merging
landline and cell
– Poststratified dual sample to age and sex.

Example 2

• You are midway through a landline survey and
You are midway through a landline survey and
want to add cell phones. You don’t know who
has a cell phone and who doesn t. What are
has a cell phone and who doesn’t What are
your options?
1) Add cell only
1) Add cell only
2) Add cell and dual‐users

Challenge 2

• Differential Nonresponse
Differential Nonresponse
• Cell‐only overrepresented when conducting
cell phone surveys.
cell phone surveys
– Contact rate
–CCooperation rate
i
• Those who rely more on their cell phone will
be easier to reach.

Telephone Reliance
C
L
e
a
l
n
l
d
l
i P
n h
e o
n
e

Landline Frame
C
L
e
a
l
n
l
d
l
i
Landline households P
n h
e o
n
e

Landline sample

Cell Phone Frame
C
L
e
a
l
n
l
d
l
i
Cell Phone Users P
n h
e o
n
e

Cell sample
Cell sample

Dual Frames

Landline sample Landline sample

Cell sample Cell sample

Dual Frame sample Dual Frame sample

Ideal Realistic

Our Goal

Rebalance
R b l
on cell
reliance

Rebalance
on landline
reliance

Measuring Telephone Reliance

• Cell only landline only
Cell only, landline only
• Classify Dual users
– “Of ll th t l h
“Of all the telephone calls that you receive, are…”
ll th t i ”
• All or almost all calls received on a cell phone? (cell‐
mostly)
• Some received on a cell phone and some on a regular
landline phone? (true‐dual)
• Very few or none received on a cell phone? (landline‐
mostly)

Telephone Reliance
C
L
e
a
l
n
d Landline Landline True Cell Cell l
l
i
Only Mostly Dual Mostly Only P
n (0) (1) (2) (3) (4) h
e o
n
e

Response Propensity

• Adjust for differential nonresponse by
Adjust for differential nonresponse by
benchmarking against NHIS
• Logistic regression model
Logistic regression model
– Dependent: Survey type
•1 b
1 = observe cell user in national cell phone survey
ll i ti l ll h
• 0 = observe cell user in NHIS
– Independent: Cell phone reliance (1 4) age race
Independent: Cell phone reliance (1‐4), age, race

Data sources

National cell sample
National cell sample NHIS
n=500

Landline mostly 8% 23%

True dual 27% 42%

Cell mostly 23% 17%

Cell only 42% 18%

Cell users 100% 100%

Applying the model

• Applied to same data—poststatification
Applied to same data poststatification
• Applied to independent data
–A
Assumption: national cell sample measures the
ti ti l ll l th
odds ratio for observing a cell‐only respondent in
a cell sample relative to a dual user.
a cell sample relative to a dual‐user

• S
State, local surveys
l l

Applying the model

Colorado
Cell Sample
w/o NR adj w/ NR adj

Landline mostly (1) 9% 26%
True dual (2) 26% 37%
Cell mostly (3)
C ll tl (3) 19% 15%
Cell only (4) 46% 21%
Total cell sample 100% 100%

Applying the model

• Assume landline only is 20% (we don’t know)
Assume landline only is 20% (we don t know)
CO

Landline only 20%

Landline mostly 21%

True dual 30%

Cell mostly 12%
NHIS
NHIS
Cell only 17% state
estimate
Total population
p p 100% = 15%

City Sample

Landline Cell Sample Combined
Sample Samples

w/o NR
w/o NR w/ NR
w/ NR w/o NR
w/o NR w/ NR
w/ NR
adj adj adj adj

Cell‐only ‐ 43.5 18.9 35.5 13.4
Cell‐mostly 12.4 25.1 19.8 13.8 12.2
True Dual 30.1 23.6 35.5 18.7 25.1
Landline mostly
Landline‐mostly 19.1 7.9 25.8 9.9 17.2
Landline‐only 38.4 ‐ ‐ 23.9 32.1

Conclusions

• Dual‐frame
Dual frame
– There are ways to combine the data, even when
we don t have a full picture of group membership.
we don’t have a full picture of group membership
• Differential nonresponse
– Response propensity model rebalances the cell
Response propensity model rebalances the cell
sample based on cell reliance.
– Can be applied at state and local levels when no
Can be applied at state and local levels when no
benchmarks exist.
– Next steps: explore a response propensity model
Next steps: explore a response propensity model
for landline.

Thank you

Randal.Zuwallack@macrointernational.com
Randal Zuwallack@macrointernational com
Frederica.Conrey@macrointernational.com

Examining the bias in landline only
g y
surveys: How does the cell phone only
population differ from the landline
population differ from the landline
population on health indicators, and are
estimates from landline surveys biased?

Daniel A. Gundersen, MA, UMDNJ‐SPH
Cristine D. Delnevo, PhD, MPH, UMDNJ‐SPH
Randy S. ZuWallack, MS, ICF Macro

Cell Phone Substitution and RDD surveys
Cell Phone Substitution and RDD surveys
• RDD surveys (e.g. BRFSS) have traditionally only sampled
household telephones (i.e. landlines)
• Up until early 2000s, rate of cell phone only households
was small
• From mid 2000s, rate of substitution has grown
substantially
– 6 7% of adults in 2005 to 18 4% in 2008 nationally1
6.7% of adults in 2005 to 18.4% in 2008 nationally
• Higher among certain demographic groups1
– Young adults
– Hispanics and Blacks
– Poor and near poor

What is bias due to coverage error in the
sampling frame?
li f ?
• Non‐covered population is different from
Non covered population is different from
covered population on some variable of
interest

– If proportion of non‐covered ( ) is small, bias will
be small
– If difference between the covered and
,
noncovered is small, bias will be small

Previous Research
Previous Research
• Data from Jan 2004‐June 2005 NHIS found2
– Greater than 1 percentage point bias in binge drinking, smoking
prevalence, usual place for medical care, receiving influenza
vaccine
• Data from 2007 NHIS found3
Data from 2007 NHIS found
– Bias increased slightly for past year binge drinking and receiving
influenza vaccine
– These biases were larger among young adults and low income
These biases were larger among young adults and low income
persons
• Data from 2001‐2005 BRFSS on 18‐24 year olds found4
– Prevalence of binge drinking, heavy drinking, and cigarette
smoking declined during 2003‐2005; coincided with large
increase in wireless substitution among young adults
– NHIS and NSDUH did not observe similar declines during this
period

Our Study:
Our Study:
• Objective:
Objective:
– Assess the presence of bias in landline RDD due to
exclusion of cell phone only on select health
exclusion of cell phone only on select health
indicators
• Data Source and Instrument:
Data Source and Instrument:
– Cell phone RDD of adults in Colorado (n=501)
• May to September 2008
May to September 2008
• Instrument was shortened version of BRFSS
– BRFSS from same data collection period (n=4,527)
BRFSS from same data collection period (n 4,527)

Methodology
• Cell Phone sample:
Ce o e sa p e:
– Design weights account for probability of selection
• BRFSS
– Standard BRFSS design weight accounts for strata,
number of landlines and adults in the household
– Postratified b
f d by age(7)*sex*race
( )* *
• Merged data
–D i
Design weights scaled to represent share of
i ht l dt t h f
population by phone status
– Postratified by age(7)*sex*race
y g ( )

Statistical Analyses
Statistical Analyses
• Comparisons of BRFSS landline and Cell Only based on design
weights
• Comparison of BRFSS landline and merged data are
postratified to demographic makeup of CO

• We assume the merged data to be unbiased (i.e. no coverage
error due to cell only exclusion)
error due to cell only exclusion)
• All analyses conducted in STATA v.10.1 to account for complex
sampling design

Table. CO BRFSS vs. CO Cell Only, May‐September 2008 (n=5,028)
y, y p ( , )

BRFSS landline Cell only
y Difference
Health Indicator % (95%CI) % (95%CI) % (95%CI)

Smoking*
* 15.29 (±1.14) 28.14 (±4.46) ‐12.85 (±4.60)
( ) ( ) ( )

Ever had HIV test* 36.64 (±1.85) 52.51 (±5.09) ‐15.87 (±5.41)

Having health insurance* 88.36 (±1.11) 72.46 (±4.38) 15.9 (±4.51)

Having primary care provider* 85.91 (±1.16) 60.39 (±4.85) 25.52 (±5.00)

Not affording care due to cost 12.27 (±1.08) 20.42 (±3.94)
Not affording care due to cost* 12 27 (±1 08) 20 42 (±3 94) ‐8.15 (±4.08)
‐8 15 (±4 08)
*p<.05; data weighted to correct for sampling design

Figure 1. Bias of Smoking Prevalence, CO BRFSS (n=4,527) and CO Cell Only
RDD (n=501) May‐September 2008
RDD (n 501) May September 2008
10

7.5
75

5

2.5

0

‐2.5

‐5

‐7.5

‐10

Figure 2. Relative Bias of Smoking Prevalence, CO BRFSS (n=4,527) and CO Cell
Only RDD (n=501) May‐September 2008
Only RDD (n 501) May September 2008

50%

40%

30%

20%

10%

0%

‐10%

‐20%

‐30%

‐40%

‐50%
50%

Figure 3. Bias of Ever had an HIV test, CO BRFSS (n=4,527) and CO Cell Only

10

7.5

5

2.5

0

‐2.5

‐5

‐7.5

‐10

Figure 4. Relative Bias of Ever had an HIV test, CO BRFSS (n=4,527) and CO
Cell Only RDD (n=501) May‐September 2008
Cell Only RDD (n 501) May September 2008

50%

40%

30%

20%

10%

0%

‐10%

‐20%

‐30%

‐40%

‐50%
50%

Figure 5. Bias ‐ Having health insurance, CO BRFSS (n=4,527) and CO Cell Only

10

7.5

5

2.5

0

‐2.5

‐5

‐7.5
‐7 5

‐10

Figure 6. Relative Bias Having Health Insurance, CO BRFSS (n=4,527) and CO

50%

40%

30%

20%

10%

0%

‐10%

‐20%

‐30%

‐40%

‐50%
50%

Figure 7. Bias Has primary care provider, CO BRFSS (n=4,527) and CO Cell Only

10

7.5

5

2.5

0

‐2.5

‐5

75
‐7.5

‐10

Figure 8. Relative Bias Has Primary Care Provider, CO BRFSS (n=4,527) and CO

50%

40%

30%

20%

10%

0%

‐10%

‐20%

‐30%

‐40%

‐50%
50%

Figure 9. Bias ‐ could not afford health care due to cost, CO BRFSS (n=4,527)
and CO Cell Only RDD (n=501) May‐September 2008
and CO Cell Only RDD (n 501) May September 2008

10

7.5

5

2.5

0

‐2.5

‐5

‐7.5
7.5

‐10

Figure 10. Relative Bias ‐ could not afford health care due to cost, CO
BRFSS (n=4,527) and CO Cell Only RDD (n=501) May‐September 2008
BRFSS (n 4 527) and CO Cell Only RDD (n 501) May September 2008

50%
40%
30%
20%
10%
0%
‐10%
‐20%
‐30%
‐40%
‐50%

Summary of Findings
Summary of Findings
• Bias is present not only among those with high wireless
substitution rates
• Smoking prevalence underestimated among those with
higher wireless substitution rates
higher wireless substitution rates
• Ever had an HIV test substantially underestimated
among all groups
– R l ti bi l
Relative bias large among those with high wireless
th ith hi h i l
substitution rates (young adults, non‐whites, low SES)
• Bias for health care insurance and having primary care
provider is underestimated among non‐whites, but
id i d i d hi b
overestimated among other groups
– Relative bias is small

Implications for study design and
analysis
l
• When possible, include an RDD of cell phone only
population (BRFSS now does this)
– If you can’t, be aware of the potential for bias and interpret
findings accordingly
• If you’re analyzing landline RDD data from past years
– Interpret findings with potential bias in mind
– Historical trend may observe artificial changes due to coverage
error
– Wireless substitution rates differ by geographic region so
problem may be less in certain areas
• A bi
A bias present today may not be the same historically or in
d b h hi i ll i
the future
– Characteristics of the early adopters may not be the same as the
current cell only population today or laggards
current cell only population today or laggards

Limitations
• Unable to assess bias in some subpopulations
Unable to assess bias in some subpopulations
due to small sample size
• Study does not account for cell phone mostly
Study does not account for cell phone mostly
population

References
1. Blumberg SJ & Julian V. Luke. (2009). Wireless Substitution: Early
release of estimates from the National Health Interview Survey,
l f ti t f th N ti l H lth I t i S
July‐December 2008.
2. Blumberg SJ, Luke JV & Marcie L. Cynamon. (2006). Telephone
coverage and health survey estimates: evaluating concern about
coverage and health survey estimates: evaluating concern about
wireless substitution. American Journal of Public Health. 96(5):
926‐931.
3. Blumberg SJ & Julian V. Luke. (2009). Reevaluating the need for
concern regarding noncoverage bias in landline surveys. American
Journal of Public Health. 99(10): 1806‐1810.
4. Delnevo CD, Gundersen DA & Brett T. Hagman. (2008). Declining
estimated prevalence of alcohol drinking and smoking among
estimated prevalence of alcohol drinking and smoking among
young adults nationally: artifacts of sample undercoverage?
American Journal of Epidemiology. 167(1): 15‐19.

Contact Info
Contact Info
Cristine Delnevo, PhD, MPH delnevo@umdnj.edu
Cristine Delnevo PhD MPH delnevo@umdnj edu
Daniel A. Gundersen, MA gunderda@umdnj.edu
Randal ZuWallack Randal.Zuwallack@macrointernational.com
Riki Conrey Frederica.Conrey@macrointernational.com

Random digit dialing cell phone surveys and surveillance systems: Assessing data quality and bias

Recommended

Recommended

More Related Content

Similar to Random digit dialing cell phone surveys and surveillance systems: Assessing data quality and bias

Similar to Random digit dialing cell phone surveys and surveillance systems: Assessing data quality and bias (20)

Recently uploaded

Recently uploaded (20)

Random digit dialing cell phone surveys and surveillance systems: Assessing data quality and bias