The most reliable source of Evidence Based Medicine.

1. META-
ANALYSIS
GAURAV KAMBOJ
Junior Resident
Deptt. of Community Medicine
PGIMS, Rohtak

2. Contents
•Introduction
•Types of reviews
•Definition
•Functions of meta-analysis
•Conducting Meta-analysis
•Writing the research question and protocol
•Comprehensive search
•Selection of studies
•Appraisal (quality assessment) of studies
•Data abstraction
•Data analysis

3. •Effect size
•Presenting the findings – Forest plot
•Heterogeneity
•Dealing with heterogeneity
•Fixed and random effects model
•Meta-regression
•Strengths and Weaknesses of meta-analysis
•Software for meta-analysis
Contents

4. INTRODUCTION

5. Rationale for reviews
Information explosion
• More than 1,00,000 articles
are published each year in
more than 20,000 journals.
• Humanly impossible to
read through the articles
published in any field.
• Concise summaries of
literature (Reviews)
required, after separating
insignificant and unsound
from salient and crucial.

7. Literature reviewing - conceptual relations
Systematic review
Meta-analysis
Narrative review

8. What is a traditional / narrative review?
“ Review articles written by one or more experts based
on a convenience sample of studies with no description
of the underlying methodology”
•Do not statistically combine results from multiple
studies
•Vote-counting

9. What is a systematic review?
“A review that has been prepared using some kind of
systematic approach to minimising biases and random
errors, and that the components of the approach will be
documented in a materials and methods section”
- Chalmers et al
• Has only qualitative or both qualitative and quantitative
components
• Quantitative component is meta-analysis

10. Meta-analysis
“Quantitative and statistical approach for
systematically combining results of
previous research to arrive at
conclusions about the body of research.”
-Glass

11. •The term meta-analysis means ‘an analysis of analysis’.
•A particular topic may have been replicated in various
ways, using, for example, differently sized samples, and
conducted in different countries under different
environmental, social and economic conditions.
•Sometimes results appear to be reasonably consistent;
others less so.
•Meta-analysis enables a rigorous comparison to be made
rather than a subjective ‘eyeballing’.
•If some crucial factors like sample size and methodology
are missing then comparison is not feasible.
What is meta-analysis?

12. The Great Debate
•1952: Hans J. Eysenck concluded that there were no
favorable effects of psychotherapy, starting a raging debate
which 25 years of evaluation research and hundreds of studies
failed to resolve
•1978: To proved Eysenck wrong, Gene V. Glass
statistically aggregated the findings of 375 psychotherapy
outcome studies
Glass (and colleague Smith) concluded that psychotherapy did
indeed work.
Glass called the method “Meta-analysis”

13. Functions of Meta-Analysis
•Identifies heterogeneity in effects among multiple
studies and, where appropriate, provide summary
measure
•Increases statistical power and precision to detect an
effect
•Develops ,refines, and tests hypothesis
•Reduces the subjectivity of study comparisons by using
systematic and explicit comparison procedure
•Identifies data gap in the knowledge base and suggest
direction for future research
•Analyses if and how previous studies have modified
knowledge on a certain topic

14. Forest plots of the meta-analysis addressing the use of antibiotic prophylaxis
compared with no treatment in colon surgery

15. Conducting
Meta-Analysis

16. Conducting a Meta-Analysis
Writing the research question and a protocol
Comprehensive search
Selection of studies
Appraisal (quality assessment) of studies
Data abstraction
Data analysis

17. Writing the research question and protocol
•Research question:
• P: the population of interest
• I: the intervention or exposure
• C: the comparison (in certain situations)
• O: the outcome of interest
•Protocol: specifying the –
• Research question
• Search methods
• Inclusion and exclusion criteria for studies
• Criteria for quality assessment (appraisal) of the studies
• Methods of data abstraction and synthesis

18. Comprehensive Search
Hand searching – ‘gold-standard’ for published studies
• Computerized databases:
• Pubmed/Medline (www.ncbi.nlm.nih.gov./entrez/query.fcgi),
• EMBASE,
• ScienceDirect , (www.sciencedirect.com)
• Scirus, (www.scirus.com/srsapp ),
• Cochrane Review- CENTRAL (Cochrane Central Register of Controlled Trials,
http://www.mrw.interscience.wiley.com/cochrane/cochrane_clcentral_articles_fs.ht
m)
• Trials Register- ClinicalTrials.gov
• Personal references, and emails
• Web e.g. ISI Web of Knowledge (http://www.isiwebofknowledge.com),
Google Scholar(http://scholar.google.com)
• Conference programs
• Dissertations
• Review articles
• Government reports, bibliographies

20. Example: Research Issue
Let's say we want to know whether
streptokinase is protective for death
from acute myocardial infarction.
How should we set up a search strategy?
We will search pubmed only

21. The Search
• “streptokinase”[text word] OR “acute myocardial infarction”
produces ALL articles that contain EITHER streptokinase OR
acute myocardial infarction anywhere in the text – inclusive,
many
• streptokinase [text word] AND “acute myocardial infarction”
[text word] will capture only those subsets that have BOTH
streptokinase AND acute myocardial infarction anywhere in the
text – restrictive, few
Next, we shall look at the PUBMED Screen …

22. Choose your DATABASE here
Remember to choose both
PUBMED, and MESH for
formulating search. Choose
PUBMED CENTRAL for free
articles!

23. Keep some, throw out others
•Keep the ones with
•high levels of evidence
•good quality
•check with QUOROM guidelines
•Usually, MetaAnalysis done with RCTs
•Case series, and case reports definitely out
Selection problems are major problems

24. Selection of studies
•Explicit Inclusion and exclusion criteria
• Study designs: RCTs or CTs with a control group
• Subjects: e.g. Females > 18 years of age
• Publication types: Journal articles, dissertations, & masters
theses
• Languages: English
• Interventions: e.g. Bone mineral density assessed at femur,
spine, and/or radius
• Time Frame: Studies published & indexed between January
1966 and December 1998

25. Factors Affecting Study Quality
Non-randomized trials:
•Treatment allocation related to prognosis or pre-
judgment of appropriateness of treatment
Randomized trials:
•Inadequate randomization (e.g. alternating assignment)
•Lack of stratification on important factors
•Lack of or ineffective blinding
All trials:
•Patient drop-outs, patient switching arms
•Missing data
•Improper statistical analysis
Appraisal of studies

26. Assessing Study Quality
• Quality scores developed by -
• Chalmers et al
• Jadad et al
• None is absolute best.
• Little is known about their relative merits and their
association with study outcomes.
• When studies are excluded from a meta-analysis, reasons for
exclusion should be provided for each excluded study
• GIGO principle of ‘garbage in, garbage out’

27. How to score the quality of a study?
•Example (scored yes=1, no=0):
• Published in a peer-reviewed journal?
• Experienced researchers?
• Research funded by impartial agency?
• Study performed by impartial researchers?
• Subjects selected randomly from a population?
• Subjects assigned randomly to treatments?
• High proportion of subjects entered and/or finished the study?
• Subjects blind to treatment?
• Data gatherers blind to treatment?
• Analysis performed blind?
• Use the score to exclude some studies, and/or…
• Include as a covariate in the meta-analysis

28. Checking for Bias
•Reporting Bias
is a group of related biases potentially leading to over-
representation of significant or positive studies in
systematic reviews
•Studies with significant positive findings -
• More likely to be published- Publication bias - over estimation of treatment
effects
• More likely to be published rapidly - Time lag bias
• More likely to be published in English - Language bias
• More likely to be cited by others - Citation bias

29. Identifying Publication Bias
•Funnel Plot
• Display the studies included in meta-analysis in a plot of effect size against sample
size (or some other measure of the extent to which the findings could be affected by
the play of chance).
• Egger’s Regression Test:
• Tests whether small studies tend to have larger effect sizes than would be expected
(implying that small studies with small effect sizes have not been published).
• Begg’s rank correlation test
Both rarely used

30. Funnel Plot: what and how to read
Plots the effect size against
the sample size of the study
To study a funnel plot, look at
its LOWER LEFT corner, that’s
where negative or null studies
are located
If EMPTY, this indicates
“PUBLICATION BIAS”
Note that here, the plot fits in
a funnel, and that the left
corner is not all that empty,
but we cannot rule out
publication bias

31. An Asymmetric Funnel Plot
(indicative of publication bias)
(Region
of missing
studies)
Log Odds Ratio
-2 -1 0 1 2
Asymmetric plot –
•Publication bias
•Clinical heterogeneity
•Methodological
heterogeneity

33. Meta-analysis in Presence of
Publication Bias
• Combine the results of larger studies only, which are less
likely subject to publication bias.
• File-drawer Method / Fail safe N: How many unpublished
studies showing a null result are required to change a
‘significant’ meta analysis result to a ‘non-significant’ one?
• ‘Trim and Fill’ method:
• Tail of the side of funnel plot with smaller studies is chopped off to
make the funnel plot symmetrical
• Replicated and added back to both sides so the plot becomes
symmetrical.
• The centre and variability of the filled funnel plot are then
estimated (there are complicated statistical methods to do this
formally).

34. An Asymmetric Funnel Plot
(indicating publication bias)
Log Odds Ratio
-2 -1 0 1 2
Trimmed
Filled
Estimated # missing
studies : 5

35. Precautions
•At least two reviewers
•Sift and sift again
• The first sift – pre-screening - is to decide which studies to retrieve in full.
• The second sift – selection - is to look again at these studies and decide which are
to be included in your review
•Do not collect outcome data at the same time as
eligibility information
• wasted time and effort - if study is excluded later on
• Results can sway decision
•Look out for duplicate publications

36. How to Abstract Data: Guidelines
Seven columns created
trial: trial identity code
trialnam: name of trial
year: year of the study
pop1: study population
deaths1: deaths in study
pop0: control population
deaths0: deaths in control
22 studies to do meta analysis

37. 1. Choice of Metric
Data Type Outcome Measures
Continuous Mean
Dichotomous (binary)
(displayed in 2x2 table)
Odds ratio (OR),
Risk ratio (RR),
Risk difference (RD)
Data analysis

38. Comparison of OR, RR, and RD
Failure Success Total
New Treatment 5 95 100
Control 10 90 100
Odds Ratio = (5/95) / (10/90) = 0.48
Risk Ratio = (5/100) / (10/100) = 0.50
(Recall OR  RR when probability is small. OR is generally more extreme
(further from 1) than RR.)
Risk Difference = (5/100) - (10/100) = -0.05

39. Effect sizes
• The effect size makes meta-analysis possible
“ratio of the frequency of the events in the intervention to
that in the control group.”
• Any standardized index can be an “effect size” (e.g.,
standardized mean difference, correlation coefficient, odds-
ratio) as long as it –
• Is comparable across studies (generally requires standardization)
• Represents the magnitude and direction of the relationship of interest
• Is independent of sample size
• Different meta-analyses may use different effect size indices
• Studies are weighted according to the inverse of their
variance.

40. gHedges 
YExperimental  YControl
((NE 1)SD2
E  (NC 1)SD2
C )) / (NTot  2)
 1
3
4(NE  NC )  9




Glass 
YExperimental  YControl
SDControl
ES Calculation: Descriptive Statistics
dCohen 
YExperimental  YControl
(SD2
E  SD2
C ) / 2

41. Zero Effect Size
ES = 0.00
Control
Group
Intervention
Group
Overlapping
Distributions

42. Moderate Effect Size
Control
Group
Treatment
Group
ES = 0.40

43. Large Effect Size
Control
Group
Intervention
Condition
ES = 0.85

44. Presenting the findings - Forest plots
•The graphical display of results from individual
studies on a common scale is a “Forest plot”.
•Each study is represented by a black square and a
horizontal line (CI:95%).
•The area of the black square reflects the weight of
the study / precision of the study (roughly the
sample size).
•A logarithmic scale should be used for plotting the
Relative Risk / Odds Ratio.
•Aggregate Effect size – displayed as a ‘diamond’.

45. Forest plot

46. Forest plot
The impact of fish oil consumption on Cardio-vascular diseases
LINE OF NO EFFECT
i.e. no statistically
significant difference
between the study
and control group

47. D'Souza, A. L et al. BMJ 2002;324:1361
Effect of probiotics on the risk of antibiotic associated diarrhoea
The label tells what
the comparison and
outcome of interest
are
Scale measuring
treatment effect.
Take care when
reading labels!
Each study
has an ID
(author)
Treatment effect
sizes for each study
(plus 95% CI)

48. Heterogeneity
Reviews usually bring together studies that were performed:
• By different people
• In different settings
• In different countries
• On different people
• In different ways
• For different lengths of time
• To look at different outcomes
Types of heterogeneity
•Clinical heterogeneity
•Methodological heterogeneity
•Statistical heterogeneity

49. Dealing with statistical heterogeneity
•Test for existence of heterogeneity: have low power
• Cochrane’s Q – statistic based on chi-square test
• I2 statistic – scores heterogeneity between 0% and 100%
• 25% - low heterogeneity
• 50% - moderate
• 75% - high
• Presence or absence of heterogeneity influences the subsequent
method of analysis:
• Fixed- effects model
• Random effect model
• Meta-regression: to over come heterogeneity

51. Assessing heterogeneity from a forest plot

52. Fixed effects model
• Conduct, if heterogeneity is absent
• Assumes the size of treatment effect
be same (fixed) across all studies &
variation due to chance
• Pooling: Mantel Haenszel OR
• Weight = 1/variance
= 1/SE2
• When heterogeneity exists we get:
• a pooled estimate which may give too
much weight to large studies,
• A narrow confidence interval
• a P-value which is too small.
Random effects model
• Conduct, if heterogeneity is present
• Assumes the size of treatment effect
does vary between studies
• Der Simonian Laird method (DSL) for
Odds’ Ratio
• Weight = 1/variance
= 1/(SE2+ inter-trial variance)
• When heterogeneity exists we get:
• a different pooled estimate with a
different interpretation,
• a wider confidence interval,
• a larger P-value

53. Fixed effects model
• When heterogeneity does not exists:
• a pooled estimate which is correct,
• a confidence interval which is correct,
• a P-value which is correct.
Random effects model
• When heterogeneity does not exist:
• a pooled estimate which is correct,
• a confidence interval which is too wide,
• a P-value which is too large
No universally accepted method for choosing.
A reasonable approach:
1. Decide whether the assumption of a fixed effects model is plausible. Could the
studies all be estimating the same effect? If not, consider a random effects model.
2. If fixed effects assumption is plausible, are the data compatible?
Graphical methods: forest plot, Galbraith plot.
Analytical methods: heterogeneity test, I2
statistic.
If assumption looks compatible with the data, use fixed effects, otherwise
consider random effects.

55. Meta-regression
• Allows researchers to explore which types of patient-specific
factors or study design factors contribute to heterogeneity.
• The estimate of study results is the dependent variable and one
or more study-level variables are the independent variables
(predictors)
• Uses summary data from each trial, such as the average effect
size, average disease severity at baseline, and average length of
follow-up.

56. Quality Assessment of MA
•PRISMA Statement (formerly QUOROM) : Preferred
Reporting Items for Systematic Reviews and Meta-
Analyses
•MOOSE Statement : proposal for reporting meta
analyses of observational studies in epidemiology

59. Strengths & Weaknesses

60. Strengths
• Comprehensive search strategy: multiple sources of
information
• Explicit methodology: to ensure reproducibility and
transparency
• Emphasis on all clinically important outcomes: related to
efficacy, safety, and tolerability of the interventions under
consideration
• Limiting errors: two reviewers at all major steps; limits bias
and improves precision

61. Weaknesses
 Good deal of effort
 Qualitative distinctions between studies not
captured
 A good meta-analysis of badly designed studies will
still result in bad statistics.
 Selection bias
 Tends to look at ‘broad questions’ that may not be
immediately applicable to individual patients
65

63. Softwares
•Huge Checklist [http://faculty.ucmerced.edu/wshadish/]
•Free Software:
•EpiMeta: from Epi Info
•Revman: from Cochrane Collaboration
•Non-free
•Meta module in STATA

64. Further reading
1. Egger M, Smith GD, Altman DG (eds). Systematic Reviews in Health Care:
Meta-analysis in context, 2nd edn. London: BMJ Publishing Group, 2001.
2. Petticrew M, Roberts H. Systematic Reviews in the Social Sciences: A
practical guide. Oxford: Blackwell Publishing, 2006.
3. Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 9: Analysing data and
undertaking meta-analyses. In: Higgins JPT, Green S (editors). Cochrane
Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated
March 2011]. The Cochrane Collaboration, 2011. Available from
www.cochrane-handbook.org.

65. THANK YOU