1. N-BIAS Nonresponse BiasAssessment Techniques CARMA Internet Research Module Jeffrey Stanton

2. May 15-17, 2008 Internet Data Collection Methods (Day 2-2) N-BIAS Methods Rogelberg, S. G, & Stanton, J. M. (2007). Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195-209.

3. May 15-17, 2008 Internet Data Collection Methods (Day 2-3) N-BIAS Methods Wave Analysis Archival Analysis Follow-up survey of non-respondents Passive non-responseanalysis (time-to-respond vs. substantive) Active non-respondentpre-study analysis Interest level analysis Worst case resistance Benchmarking Triangulation Detect, estimate, and optionally compensate for the presence, direction, and magnitude of non-response bias. Gives clearer picture of extent of bias but few good options for eliminating it.

4. May 15-17, 2008 Internet Data Collection Methods (Day 2-4) N-BIAS Methods N-BIAS comprises eight techniques Archival Analysis Follow-up Approach Wave Analysis Passive Nonresponse Analysis Interest Level Analysis Active Nonresponse Analysis Worst Case Resistance Demonstrate Generalizability

5. May 15-17, 2008 Internet Data Collection Methods (Day 2-5) Technique 1: Archival Analysis Most common technique The researcher identifies an archival database that contains the members of the whole survey sample (e.g. personnel records). That data set, usually containing demographic data, can be described: 50% Female; 40% Supervisors, etc After data collection, code numbers on the returned surveys (or access passwords) can be used to identify respondents, and by extension nonrespondents. Using this information, the archival database can be partitioned into two segments: 1) data concerning respondents; and 2) data concerning nonrespondents.

6. May 15-17, 2008 Internet Data Collection Methods (Day 2-6) So, if you found the above do you have nonresponse bias?

7. May 15-17, 2008 Internet Data Collection Methods (Day 2-7) Technique 2: Follow-up Approach Using identifiers attached to returned surveys (or access passwords), respondents can be identified and by extension nonrespondents. The follow-up approach involves randomly selecting and resurveying a small segment of nonrespondents often by phone. The full or abridged survey is then administered. In the absence of identifiers, telephone a small random sample and ask whether they responded or not to the initial survey. Follow-up with survey relevant questions

8. May 15-17, 2008 Internet Data Collection Methods (Day 2-8) Technique 3: Wave Analysis By noting in the data set whether each survey was returned before the deadline, after an initial reminder note, after the deadline, and so on, responses from pre-deadline surveys can be compared with the late responders on actual survey variables (e.g. compare job satisfaction levels).

9. May 15-17, 2008 Internet Data Collection Methods (Day 2-9) Technique 4: Passive Nonresponse Analysis Rogelberg et al. (2003) found that the vast majority of nonresponse can be classified as being passive in nature (approx. 85%). Passive nonresponse does not appear to be planned. When asked (upon receipt of the survey), these individuals indicate a general willingness to complete the survey – if they have the time. Given this, it is not surprising that they generally do not differ from respondents with regard to job satisfaction or related variables.

10. May 15-17, 2008 Internet Data Collection Methods (Day 2-10) Technique 5: Interest Level Analysis Researchers have repeatedly identified that interest level in the survey topic is one of the best predictors of a respondent’s likelihood of completing the survey. As a result, if interest level is related to attitudinal standing on the topics making up the survey, the survey results are susceptible to bias. E.g., if low interest individuals tend to be more dissatisfied on the survey constructs in question, results will be biased “high”

11. May 15-17, 2008 Internet Data Collection Methods (Day 2-11) Technique 6: Active Nonresponse Analysis Active nonrespondents, in contrast to passive nonrespondents, are those that overtly choose not to respond to a survey effort. The nonresponse is volitional and a priori (i.e. it occurs when initially confronted with a survey solicitation). Active nonrespondents tend to differ from respondents on a number of dimensions typically relevant to the organizational survey researcher (e.g. job satisfaction)

12. May 15-17, 2008 Internet Data Collection Methods (Day 2-12) Technique 7: Worst Case Resistance Given the data collected from study respondents in an actual study, one can empirically answer the question of what proportion of nonrespondents would have to exhibit the opposite pattern of responding to adversely influence sample results. Similar philosophy as what occurs in meta-analyses when considering the “file-drawer problem” By adding simulated data to an existing data set, one can explore how resistant the dataset is to worst case responses from non-respondents.

13. Technique 8 Benchmarking Using measures with norms for the population under examination, compare means and standard deviations of the collected sample to the norms

14. May 15-17, 2008 Internet Data Collection Methods (Day 2-14) Technique 9: Demonstrate Generalizability By definition, nonresponse bias is a phenomenon that is peculiar to a given sample under particular study conditions. Triangulating with a sample collected using a different method, or varying the conditions under which the study is conducted should also have effects on the composition of the nonrespondents group.

15. May 15-17, 2008 Internet Data Collection Methods (Day 2-15) Exercise 2: Item Writing Write three Likert scaled items (each) to tap into the following response-related constructs: Busyness (how busy the respondent feels) Topical interest (how interested the respondent is in the topic of the survey) Satisfaction with the sponsoring entity (the group who is running the survey) Response modality preference (e.g., got paper survey but preferred web)

16. May 15-17, 2008 Internet Data Collection Methods (Day 2-16) Establish Generalizability Demographic Profile: 1) Archival analysis Attitudinal Profile: 1) Follow-up approach 2) Wave analysis Warning Signs: 1) Interest level analysis 2) Passive NR analysis 3) Active NR analysis 4) Worst-case analysis

17. May 15-17, 2008 Internet Data Collection Methods (Day 2-17) Important Footnote In many of the N-BIAS techniques described above, one is amassing evidence for the absence of non-response bias by making comparisons and tests that preferably lead to non-significant statistical results. What is the problem with this?