1. 1Department of
SOCIAL MEDICINE
University of
BRISTOL
Risk of bias assessments:
analysis and interpretation
Jonathan Sterne
Department of Social Medicine, University of Bristol, UK

2. 2Department of
SOCIAL MEDICINE
University of
BRISTOL
Bias in the results of meta-analyses
• Reporting biases
– Publication bias
– Selective reporting of outcomes
– Fertile ground for statistical tests
– Now, finally, addressed in the obvious way, by
mandatory trial registration

3. 3Department of
SOCIAL MEDICINE
University of
BRISTOL

4. 4Department of
SOCIAL MEDICINE
University of
BRISTOL
Bias in the results of meta-analyses
• Reporting biases
– Publication bias
– Selective reporting of outcomes
– Fertile ground for statistical tests
– Now, finally, addressed in the obvious way, by
mandatory trial registration
• Biases resulting from flaws in trial conduct

5. 5Department of
SOCIAL MEDICINE
University of
BRISTOL
5
Randomised Controlled Trials (RCTs)
• Simple idea … Eligible participants
Randomise
Intervention
group
Comparison
group
Assess
outcomes
Assess
outcomes

6. 6Department of
SOCIAL MEDICINE
University of
BRISTOL
6
• Deceptively simple idea Eligible participants
Randomise
Intervention
group
Comparison
group
Assess
outcomes
Assess
outcomes
Bias can be
introduced at
all stages of
the conduct
of RCTs
Randomised Controlled Trials (RCTs)

7. 7Department of
SOCIAL MEDICINE
University of
BRISTOL
Flaws in the conduct of RCTs
• Trials provide causal inferences about the effect of the
intervention if we randomise sufficient individuals and avoid
selection and performance biases
• This can be undermined by:
– Inadequate generation of randomisation sequence

8. 8Department of
SOCIAL MEDICINE
University of
BRISTOL
Flaws in the conduct of RCTs
• Trials provide causal inferences about the effect of the
intervention if we randomise sufficient individuals and avoid
selection and performance biases
• This can be undermined by:
– Inadequate generation of randomisation sequence
– Inadequate concealment of allocation
Problems with randomisation may
cause selection bias, if participants or
healthcare providers can predict
treatment allocation

9. 9Department of
SOCIAL MEDICINE
University of
BRISTOL
Flaws in the conduct of RCTs
• Trials provide causal inferences about the effect of the
intervention if we randomise sufficient individuals and avoid
selection and performance biases
• This can be undermined by:
– Inadequate generation of randomisation sequence
– Inadequate concealment of allocation
– Inadequate blinding
• Performance bias
• Care of intervention and control groups
not comparable
• Detection bias
• Measurement of outcomes not
comparable

10. 10Department of
SOCIAL MEDICINE
University of
BRISTOL
Flaws in the conduct of RCTs
• Trials provide causal inferences about the effect of the
intervention if we randomise sufficient individuals and avoid
selection and performance biases
• This can be undermined by:
– Inadequate generation of randomisation sequence
– Inadequate concealment of allocation
– Inadequate blinding
– Excluding patients, or analysing them in the wrong group

11. 11Department of
SOCIAL MEDICINE
University of
BRISTOL
Including biased trials will cause meta-
analyses to be biased

12. 12Department of
SOCIAL MEDICINE
University of
BRISTOL
0.2 0.4 0.6 21
Meta-analysis
Single large trial
Odds Ratio
Nitrates in myocardial infarction
Inpatient geriatric assessment
Magnesium in myocardial infarction
Aspirin for pre-eclampsia prevention
Intervention
Egger et al. BMJ 1997

13. 13Department of
SOCIAL MEDICINE
University of
BRISTOL
Including biased trials will cause meta-
analyses to be biased
• An obvious solution is to score the quality of trials included
in the meta-analysis
• We could then downweight low quality trials, or exclude
trials scoring below a chosen quality threshold

14. 14Department of
SOCIAL MEDICINE
University of
BRISTOL
The death of quality scores
• 25 known checklists
• Between 3 and 34 components
• Frequently no definitions of quality
• Most components said to be based on “accepted criteria”
(Moher et al. Controlled Clinical Trials 1995; 16: 62-73)

15. 15Department of
SOCIAL MEDICINE
University of
BRISTOL
“Quality scores are useless
and potentially misleading”
Greenland Am.J.Epidemiol. 1994;140:290-296
“perhaps the most insidious form of subjectivity
masquerading as objectivity is ‘quality scoring’. This
practice subjectively merges objective information with
arbitrary judgements in a manner that can obscure
important sources of heterogeneity among study
results”

16. 16Department of
SOCIAL MEDICINE
University of
BRISTOL
Relative risk with 95% CI
0·5 0·75 1 1·25
QualityScale1-25
All trials
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Re-analysis using 25 different
scales to assess trial quality
Endpoint: Deep vein thrombosis
Jüni et al. JAMA 1999 282: 1054-1060
High quality
Low quality

17. 17Department of
SOCIAL MEDICINE
University of
BRISTOL
Empirical evidence of bias
Evidence-based critical appraisal

18. 18Department of
SOCIAL MEDICINE
University of
BRISTOL
Meta-epidemiology
(Naylor, BMJ 1997; 315: 617-619)
• Identify a large number of meta-analyses
• Record characteristics of individual studies (allocation
concealment, blinding, type of publication, language etc.)
• Compare treatment effects within each meta-analysis (for
example not double blind vs. double blind)
• Estimate ratio of odds ratios

19. 19Department of
SOCIAL MEDICINE
University of
BRISTOL
Schulz KF, Chalmers I, Hayes RJ, Altman DG. (1995) Empirical evidence of bias. Dimensions of
methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273: 408-412.
0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2
No
Yes
Double-blind
Yes
No
Exclusions
Inadequate/unclear
Adequate
Sequence generation
Inadequate/unclear
Adequate
Concealment of allocation
Ratio of Odds Ratios
Treatment effect
over-estimation under-estimation
Reference
Empirical evidence of bias
33 meta-analyses, 250 RCTs

20. 20Department of
SOCIAL MEDICINE
University of
BRISTOL
NOTE: Weights are from random effects analysis
Overall (I-squared = 88.6%, p = 0.000)
Kjaergard, 2001
Als-Neilsen, 2004
Moher, 1998
Study ID
Balk, 2002
Egger, 2003
Schulz, 1995
0.79 (0.66, 0.95)
0.60 (0.37, 0.97)
1.02 (0.93, 1.13)
0.63 (0.45, 0.88)
ratios (95% CI)
0.95 (0.83, 1.09)
0.79 (0.70, 0.89)
0.66 (0.59, 0.73)
0.79 (0.66, 0.95)
0.60 (0.37, 0.97)
1.02 (0.93, 1.13)
0.63 (0.45, 0.88)
ratios (95% CI)
Ratio of odds
0.95 (0.83, 1.09)
0.79 (0.70, 0.89)
0.66 (0.59, 0.73)
.25 1 4.25 .5 1 2
Ratio of odds ratios
Allocation concealment:
combined evidence

21. 21Department of
SOCIAL MEDICINE
University of
BRISTOL
Analysis of meta-epidemiological
studies
• Most studies used a logistic regression approach assuming
fixed effects within and between meta-analyses
– Assumes no between-trial heterogeneity, and that effects of bias are
the same in each meta-analysis
• Two-stage approach:
– Estimate the effect trials characteristics separately in each meta-
analysis
– Combine estimates across meta-analyses
(Sterne et al. Statistics in Medicine 2002; 21: 1513-1524)

22. 22Department of
SOCIAL MEDICINE
University of
BRISTOL
Non blinded
more beneficial
Non blinded
less beneficial
0.93 (0.83, 1.04)
Ratio of odds ratios
(95% CI)
Overall (76)
No. of trials*
314vs. 432
Comparison
(No. of meta-analyses)
* Non blinded vs. blinded
Variability in
bias (P value)
0.11 (p<0.001)
The
image
cannot
be
displa…
Ratio of odds ratios
0.5 0.75 1 1.5 2
1.01 (0.92, 1.10)
0.75 (0.61, 0.93)Subjective outcomes (32)
Objective outcomes (44) 210vs. 227
104vs. 205
0.08 (p<0.001)
0.14 (p=0.001)T
h
The image
cannot be
displayed.
Your
computer may
not have
enough me…
Combined analysis of three empirical
studies: blinding

23. 23Department of
SOCIAL MEDICINE
University of
BRISTOL
Inadequately concealed
more beneficial
Inadequately concealed
less beneficial
Ratio of odds ratios
0.5 0.75 1 1.5 2
Ratio of odds ratios
(95% CI)
0.83 (0.74, 0.93)Overall (102)
No. of trials*
532 vs. 272
Comparison
(No. of meta-analyses)
* Inadequately/unclearly
concealed vs. adequately
concealed
0.11 (p<0.001)
The
image
cannot
be
displa…
The image
cannot be
displayed.
Your
computer
may not
have enou
0.91 (0.80, 1.03)
0.69 (0.59, 0.82)
Objective outcomes (62)
Subjective outcomes (40)
310 vs. 174
222 vs. 98
0.11 (p<0.001)
0.07 (p=0.011)
Variability in
bias (P value)
Combined analysis of three empirical
studies: allocation concealment
Wood, L., Egger, M., Gluud, L.L., Schulz, K., Jüni, P., Altman, D.G., Gluud, C., Martin, R.M., Wood, A.J.G.
and Sterne, J.A.C. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with
different interventions and outcomes: meta-epidemiological study. BMJ, 336: 601-605.

24. 24Department of
SOCIAL MEDICINE
University of
BRISTOL
Inadequately concealed
more beneficial
Inadequately concealed
less beneficial
Ratio of odds ratios
0.5 0.75 1 1.5 2
Ratio of odds ratios
(95% CI)
0.83 (0.74, 0.93)Overall (102)
No. of trials*
532 vs. 272
Comparison
(No. of meta-analyses)
* Inadequately/unclearly
concealed vs. adequately
concealed
0.11 (p<0.001)
The
image
cannot
be
displa…
The image
cannot be
displayed.
Your
computer
may not
have enou
0.91 (0.80, 1.03)
0.69 (0.59, 0.82)
Objective outcomes (62)
Subjective outcomes (40)
310 vs. 174
222 vs. 98
0.11 (p<0.001)
0.07 (p=0.011)
Variability in
bias (P value)
Combined analysis of three empirical
studies: allocation concealment
Wood, L., Egger, M., Gluud, L.L., Schulz, K., Jüni, P., Altman, D.G., Gluud, C., Martin, R.M., Wood, A.J.G.
and Sterne, J.A.C. (2008) Empirical evidence of bias in treatment effect estimates in controlled trials with
different interventions and outcomes: meta-epidemiological study. BMJ, 336: 601-605.
Emprical evidence on the
effect of flaws in trial
conduct will always be
limited by imperfect
reporting of trials

25. 25Department of
SOCIAL MEDICINE
University of
BRISTOL
Bias assessment in Cochrane reviews
• Project led by Julian Higgins and Doug Altman
• “Risk of bias”, not “quality”
• Cochrane reviewers are now asked to judge whether there
is a risk of bias in the results of the trial
– “Yes”: high risk of bias
– “No”: low risk of bias
1. Sequence generation (randomisation)
2. Allocation concealment
3. Blinding of participants, personnel and outcomes
4. Incomplete outcome data (attrition and exclusions)
5. Selective outcome reporting
6. Other (including topic-specific, design-specific)

26. 26Department of
SOCIAL MEDICINE
University of
BRISTOL
26
The ROB tool: how to assess items
Two components
1. Description of what happened
• possibly including ‘done’, ‘probably done’, ‘probably not done’ or
‘not done’ for some items
2. Review authors’ judgement
• whether bias unlikely to be introduced through this item (Yes, No,
Unclear)
Yes = Low risk of bias
No = High risk of bias
Unclear = unable to make a clear judgement
‘Blinding’ and ‘Incomplete outcome data’ may need
separate assessments for different outcomes

27. 27Department of
SOCIAL MEDICINE
University of
BRISTOL
27
Risk of bias table
Study: Fisman
1981
Authors’
judgement
Description
Sequence
adequately
generated?
UNCLEAR “Patients were randomly allocated”.
Allocation
concealed?
UNCLEAR No information.
Blinding?
All included
outcomes
Low risk
“double blind design”.
“Millet… resembles lecithin in appearance… When
ground, each substance could be distinguished
from the other by hue and taste but staff were not
informed as too which was which.”
Incomplete
outcome data
addressed?
All included
outcomes
High risk
Data unavailable for meta-analysis.
Randomised: lecithin = Not stated, placebo = Not
stated, Total = 33.
Missing: lecithin = 7 (non-cooperation or diarrhoea
= 2; moved to nursing home = 4, death = 2),
placebo = 5 (non-cooperation or diarrhoea = 3,
death = 2), total missing = 36%.

28. 28Department of
SOCIAL MEDICINE
University of
BRISTOL
28
Study: Fisman
1981
Authors’
judgement
Description
Free of selective
reporting?
High risk
No quantitative results reported due to lack of
effect.
It is apparently clear which outcomes were
measured.
Free of other bias? Low risk No problems apparent.
Risk of bias table

29. 29Department of
SOCIAL MEDICINE
University of
BRISTOL
Adequate sequence generation
Allocation concealment
Blinding (Patient-reported outcomes)
Blinding (Mortality)
Incomplete outcome data addressed (Short-term outcomes (2-6 wks))
Incomplete outcome data addressed (Longer-term outcomes (> 6ks))
Free of selective reporting
Free of other bias
Yes (Low risk of bias) Unclear No (High risk of bias)
Summary of risk of bias for included trials
Incorporating bias assessments into
reviews
So now I’ve dealt with bias in my review? No!

30. 30Department of
SOCIAL MEDICINE
University of
BRISTOL
30
Summarising risk of bias
• Reviewers will need to do this:
– for an outcome within a study (across bias domains)
– for an outcome across studies (for a meta-analysis)
• Outcome-level summaries should inform the
choice of meta-analytic strategy for that outcome
• Meta-analysis-level summaries should inform the
interpretation (summary of findings)

31. 31Department of
SOCIAL MEDICINE
University of
BRISTOL
Incorporating outcome-level bias
assessments into meta-analyses
1. Present all studies and provide a narrative discussion
of risk of bias
– such an approach is discouraged because
• Descriptions of bias are likely to be lost in discussion and conclusions
• Results from studies at high risk of bias should be downweighted
2. Primary analysis restricted to studies at low (or low
and unclear) risk of bias
– Often only a small proportion of trials
– Reviewers reluctant to discard information,
3. Present multiple analyses with equal prominence
– Confusing for readers and decision-makers

32. 32Department of
SOCIAL MEDICINE
University of
BRISTOL
Evaluation of use of ROB tool

33. 33Department of
SOCIAL MEDICINE
University of
BRISTOL
Testing for bias within a meta-analysis
is unlikely to help

34. 34Department of
SOCIAL MEDICINE
University of
BRISTOLOdds ratio
.01 .1 1 10 100
No. of events
Treatment Control
Australia 1976 11/33 17/31
Europe 1974 6/110 8/113
Europe 1984 8/39 15/40
Germany 1989 2/16 2/16
Germany 1994 1/18 1/18
Hong Kong 1974 1/20 1/20
Japan 1977 4/47 0/41
Switzerland 1975 1/18 3/22
Taiwan 1997 2/21 2/19
USA 1979a 6/16 11/15
USA 1979b 4/7 5/8
USA 1987 27/75 36/76
USA 1994a 2/21 0/20
USA 1994d 6/25 1/14
USA 1996a 46/136 58/89
USA 1997 88/205157/218
Subtotal 215/807317/760
Canada 1977 6/22 9/28
Romania 1976 1/20 0/20
USA 1988 15/126 18/142
USA 1994b 12/37 21/34
USA 1996b 3/10 1/11
Subtotal 37/215 49/235
Overall 252/1022366/995
Example: Clozapine versus neuroleptic
medication for schizophrenia
Concealment inadequate (H)
Concealment adequate (L)

35. 35Department of
SOCIAL MEDICINE
University of
BRISTOL
Concealment adequate (L)
Concealment inadequate (H)
All
0.25 0.50 1.00 2.00
Treatment odds ratio (log scale)
Example: Clozapine versus neuroleptic
medication for schizophrenia
ROR: 0.66 (0.31,1.41)

36. 36Department of
SOCIAL MEDICINE
University of
BRISTOLRatio (inadeqate v adequate concealment of allocation)
.01 .05 .1 .5 1 2 5 10
Combined
The effects of components
of trial quality are usually
imprecisely estimated in a
single meta-analysis
Little hope of adjusting for
the effects of trial quality
using only the information
available in the meta-
analysis
Data from Schulz et
al. (JAMA 1995)

37. 37Department of
SOCIAL MEDICINE
University of
BRISTOL
Effects of flaws in the conduct of trials
• Change in average intervention effect (bias)
– the focus of most previous research
• Between-meta-analysis variability in average
effect of bias
• Increases in between-trial heterogeneity
• If we knew that lack of blinding always
exaggerated intervention effects by 20% there
would be no problem
• Bias matters because its effects are uncertain

38. 38Department of
SOCIAL MEDICINE
University of
BRISTOL
Consequences of flaws in trial conduct

39. 39Department of
SOCIAL MEDICINE
University of
BRISTOL
Consequences of flaws in trial conduct
How might we use evidence about the
effects of flaws in trial conduct, from meta-
epidemiological studies, to combine data
from studies at high and low risk of bias in
meta-analyses?

40. 40Department of
SOCIAL MEDICINE
University of
BRISTOL
Consequences for a single study at
high risk of bias
• Correct the estimated effect of intervention, for the
average bias associated with the flaw(s) in the trial
• Increase the trial variance by adding:
– the average increase in between-trial heterogeneity
– the between-meta-analysis variance in average bias
• So, trials at high risk of bias should be
downweighted in meta-analyses

41. 41Department of
SOCIAL MEDICINE
University of
BRISTOL
Concealment adequate (L)
Concealment inadequate (H)
All
Bias-adjusted
0.25 0.50 1.00 2.00
Treatment odds ratio (log scale)
Example: Clozapine versus neuroleptic
medication for schizophrenia
Relative weight
(type H studies)=81%
Relative weight
(type H studies)=37%
w=0.41
ROR: 0.66 (0.31,1.41)

42. 42Department of
SOCIAL MEDICINE
University of
BRISTOL
Consequences for meta-analyses
• Limits to the informational value of studies at high
risk of bias:
1. Even a very large study at high risk of bias has
minimum variance corresponding to the sum of
variances of increase in heterogeneity and variance in
bias
2. Even a meta-analysis of large studies at high risk of
bias has minimum variance corresponding to the
between-meta-analysis variance in average bias
Given current knowledge, the best approach for Cochrane
review authors is to restrict meta-analyses to studies at
low (or low and unclear) risk of bias

43. 43Department of
SOCIAL MEDICINE
University of
BRISTOL
Conclusions
• Flaws in the conduct of randomized controlled trials are
important because they increase uncertainty
• If we want to include potentially biased evidence in a
systematic review, then we should downweight and correct
for bias, based on empirical evidence on its effects
• The best currently available approach for Cochrane
reviewers is to present a primary meta-analysis restricted to
studies at low (or low and unclear) risk of bias
– Bias assessments based on the Risk of Bias Tool seem widely
accepted
– Improvements to the Handbook, RevMan and training materials may
be needed to improve incorporation of bias assessments in reviews
and SoF