This document discusses sources of bias in medical research and means to assess bias. It acknowledges the Cochrane Collaboration's Bias Methods Group and provides an overview of the impact of bias. Sources of bias can occur in the production and dissemination of evidence, including reporting biases like publication bias. Meta-epidemiological studies have found empirical evidence of biases in randomized controlled trials. Methods have been developed to assess bias in primary studies. While registration of clinical trials and systematic review protocols are attempts to minimize bias, bias remains an issue and further efforts are still needed.

1. Handling Bias in Medical Research

2. Acknowledge
• Cochrane Collaboration’s Bias Methods Group

3. Overview
• Impact of bias in medical research
• Introduction to bias and systematic error
• Sources of bias in the production and dissemination of evidence
• Meta-epidemiology and empirical evidence of bias
• Means to assess bias in primary studies
• How does this lead to good medical research?

4. Potential Impact of bias
The impact that biased estimates of effect have on research is
perhaps obvious but should not be understated
Bias leads to:
Inaccurate estimate of benefit and harms for an intervention
Biases can influence the results of secondary research
systematic reviews
Both bias and “spin” can directly influence what care patients
receive and contribute to quality of care

5. What is bias?
• A bias is a systematic error, or deviation from the truth,
in results or inferences
If you were to repeat the study over and over again
the results would always be inaccurate
• Biases can vary in magnitude and direction
• There are no current, well accepted means to estimate
magnitude of bias

6. Common misconceptions
• Bias may be distinguished from methodological quality
• Although the highest standards of conduct may have
been implemented the results still may not be believable
• Bias should not be confused with impression due to
random error (sampling variation)
In systematic reviews the quality of reporting of trial
reports is fundamental to assessing potential biases
(www.consort-statement.org)

7. Published
Conceived
Performed
Submitted
Cited
Research agenda bias
Sources of bias in the production and
dissemination of evidence

8. Sackett J Chron Dis 1979
Published
Conceived
Performed
Submitted
Cited
Sources of bias in the production and
dissemination of evidence
Methodological quality
Attrition bias Misclassification bias
Berkson bias Missing clinical data bias
Confounding bias Non blind diagnosis bias
Centripetal bias Non blind test interpretation bias
Diagnostic access bias Non-respondent bias
Diagnostic purity bias Partial verification bias
Diagnostic suspicion bias Performance bias
Diagnostic vogue bias Prevalence-incidence bias
Exposure suspicion bias Recall bias
Family information bias Referral filter bias
Interviewer bias Selection bias
Membership bias Unmasking bias

9. Published
Conceived
Performed
Submitted
Cited
Sources of bias in the production and
dissemination of evidence
Reporting Biases:
Publication Bias
Time Lag Bias
Language Bias
Multiple Publication Bias
Outcome reporting bias
Citation bias

10. Further evidence of the presence of bias
In 2010, 235 biases were mapped and characterised
17,265,924 PubMed items were clustered by biases and other text, use
of terms were mapped over time
Forty bias terms were used in the title or abstract of more than 100
articles each;
confounding, response bias, sampling bias, observer bias, cognitive
bias, race bias and perceptual bias all occur in the literature before
1965;
There were shown to be bias ‘cliques’, co-appearing biases and the
evolution of terminology was mapped over time
Chavalarias, Ioannidis 2010, J Clin Epi

11. 0.2 0.4 0.6 0.8 21
Meta-analysis
Single large trial
Odds Ratio
(95% Confidence Intervals)
Nitrates in myocardial infarction
Inpatient geriatric assessment
Magnesium in myocardial infarction
Aspirin for prevention of
pre-eclampsia
Intervention:
Egger BMJ 1997
Differences are apparent between trials

12. Meta-epidemiology
• Identify a large number of meta-analyses
• Record characteristic(s) of individual studies (e.g., degree of allocation
concealment, type of publication, language etc.)
• Compare treatment effects within each meta-analysis (e.g., adequate
allocation concealment versus inadequate)
• Estimate ratio of odds ratios comparing high quality and low quality trials
(“meta-meta-analysis”)
(Naylor, BMJ 1997; 315: 617-619

13. Meta-epidemiological studies: meta-meta-analytic method
Pooled ORs calculated for trials with and
without methodological flaw within each meta-
analysis, and difference calculated as Ratio of
ORs (ROR):
ROR = OR with flaw / OR without flaw
The RORs are then meta-analysed to estimate
overall ROR across all meta-analyses and the
between MA variability in bias
Flawed studies show more benefit Flawed studies show
less benefit
ROR
ROR (95% CI)
ROR
ROR

14. Empirical evidence of bias: RCTs
Schulz JAMA 1995
0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2
No
Yes
Double-blind
Yes
No
Exclusions
Inadequate/unclear
Adequate
Sequence generation
Inadequate/unclear
Adequate
Concealment of allocation
Ratio of Odds Ratios (95% CI)
Treatment effect
over-estimation under-estimation
(favoring experimental intervention)
Reference
33 meta-analyses,
250 RCTs

15. Prominent biases in RCTs
Bias Components Evidence
Selection bias Sequence generation
Allocation Concealment
Schulz 1995, Moher 1998,
Kjaergard 2001, Siersma 2007;
Schulz 1995a Jüni 2001
Performance bias Blinding of participants and personnel Pildal 2007, Wood 2008, Boutron
2006
Detection bias Blinding of outcome assessment Pildal 2007, Wood 2008, Boutron
2006
Attrition bias Incomplete outcome data Tierney 2005, Melander 2003,
Porta 2007
Reporting bias Selective reporting
Publication bias
Hutton 2000, Hahn 2002,
Williamson 2005, Chan 2004,
Dwan 2011
Rosenthal 1979, Sterling 1959,
Scherer 2007, Easterbrook 1991,
Dickersin 1993, Hopewell 2008

16. Methods to assess bias in primary studies
Cochrane Risk of Bias Tool (RCTs)
Downs and Black (NRS)
Newcastle- Ottawa Scale (NRS)
MINORS
NIH criteria
Zaza et al.
QUADAS (DTA)
PEDro
Oxford Centre of evidence-based medicine
Deschartres 2011 J Clin Epi

17. Single-centre versus multi-centre trials
• New potential (meta)bias (Dechartres, A. 2011)
26% larger effect size in single-centre trials
(adjusting for trial size)
• “Is conduct at a single-centre a surrogate for a process measure that
would explain the result?”
• But which class of studies provides the “truth” ?
• Could fall under the auspices of reporting bias:
Single centre trials published less frequently
More room to shape results positively in small teams

18. A new concept: Meta-bias
“Meta-epidemiological studies, examining many meta-
analyses on diverse topics, have been finding risk factors
for bias in those meta-analyses-
both in the methods that meta-analysts use and in the
characteristics of studies included-
that are not clearly associated with a bias producing
process in an individual study
We are calling the differences associated with such non-
process-related factors a meta-bias.”
(Goodman and Dickersin, 2011)

19. Publication Bias
• This concept is not new (Sterling, TD. 1959)
• Well accepted and more recent processes for assessing
asymmetry of funnel plots (Sterne, JAC. 2011)
• Impacts systematic review directly
• Impacts clinical practice guidelines indirectly
• Difficult to assess when k < 5

20. McAuley 2000
Egger 2003
Published vs. unpublished
0.91 (0.85 to 0.97)
Ratio of odds ratios
0.4 0.5 0.6 0.7 0.8 0.9 1 1.2 1.4 1.6 1.8 2
Impact of publication bias

21. Funnel plots
• If all studies come from a single underlying
population, this graph should look like an
inverted funnel, with the effect sizes homing in
on the true underlying value as n increases.
Light RJ, Pillemer DB. Summing up. The science of reviewing research. Harvard University Press, 1984.

22. Funnel plot: no evidence of bias
StandardError
Odds ratio
0.1 0.3 1 3
3
2
1
0
100.6

23. Funnel plots
• In the presence of publication bias some studies
are missing
• asymmetry is present
Light RJ, Pillemer DB. Summing up. The science of reviewing research. Harvard University Press, 1984.

24. 0.1 0.3 1 3 100.6
Asymmetrical
Funnel Plot
Reporting bias present
Odds ratio
StandardError
3
2
1
0

25. “Small study effect”
There is a tendency for smaller trials in a
meta-analysis to show greater treatment
effects than the larger trials
Small study effects need not result from bias
Sterne et al. Journal of Clinical Epidemiology 2000

26. Selective reporting
• This concept is not new (Hemminki 1980)
• Recent evidence that this remains a serious issue (Kirkham et
al PLoS ONE 2010)
• New reviews from 3 consecutive issues Cochrane Library
• 22% of 288 review/protocol pairings were discrepant in at
least one outcome
• Currently included in Cochrane RoB Tool;
• Limited by the need to review trial protocols, the potential
solution is registration mandated by funding agencies

27. Global attempts to minimise bias:
Trial registration
In 2004 the International committee of Medical Journal
Editors (ICMJE) announced that they will no longer publish
trials that were not registered at inception

28. • Clinical trials.gov
• WHO portal

29. Global attempts to minimise bias:
Trial registration
This is a great start, but with the evidence mounting
suggesting that registration is not optimal

30. Registering Systematic Review Protocols:
An International Initiative
Vancouver, BC, Canada
February 18, 2011

31. • http://www.crd.york.ac.uk/prospero/
• Launched Feb 2011
• 400+ registered protocols from 27 countries

32. Thank you for your attention
David Moher: dmoher@ohri.ca