2. HOW TO CONDUCT A
SYSTEMATIC REVIEW?
DR. NIDHI PRUTHI SHUKLA
PROFESSOR & HEAD
Department of Public Health Dentistry
Rama Dental College, Hospital & Research Centre,
Kanpur, India
drnidhi_21@yahoo.in
3. “It is surely a great
criticism of our profession
that we have not
organized a critical
summary, by specialty or
subspecialty, updated
periodically, of all relevant
randomized controlled
trials.”
- Archie Cochrane
4. What is a systematic review?
SYSTEMATIC: Done or acting according to a fixed plan
or system: methodical
REVIEW: A critical appraisal of a book, play or other
work
5. What is a systematic review?
• “The application of strategies that limit bias in
the assembly, critical appraisal, and synthesis of
all relevant studies on a specific topic.”
Oxford Centre of Evidence Based Medicine (OCEBM) Levels Table
• Ensures that all available evidence is taken into
account and minimises ‘cherry-picking’
• Not performing SRs can be dangerous and/or
unethical!
6. Practising EBM – the 4 A’s
Ask a
clinical
question
Acquire
the best
evidence
Appraise
the
evidence
Apply
the
evidence
Step 1
Step 2
Step 4
Step 3
WHY WE NEED SYSTEMATIC REVIEWS
10. Key elements of a systematic review
Structured, systematic process involving several
steps :
1. Formulate the question
2. Plan the review
3. Comprehensive search
4. Unbiased selection and abstraction process
5. Critical appraisal of data
6. Synthesis of data (may include meta-analysis)
7. Interpretation of results
All steps described explicitly in the review
12. Questions of interest
Effectiveness:
• Does the intervention work/not work?
• Who does it work/not work for?
Other important questions:
• How does the intervention work?
• Is the intervention appropriate?
• Is the intervention feasible?
• Is the intervention and comparison relevant?
14. A PICO question
What is the best strategy to prevent smoking in
young people?
Time-consuming question
15. An answerable question
Are mass media (or school-based or
community-based) interventions effective in
preventing smoking in young people?
16. Problem,
population
Intervention Comparison Outcome Types of
studies
Young people
under 25 years
of age
a) Television
b) Radio
c) Newspapers
d) Bill boards
e) Posters
f) Leaflets
g) Booklets
a) School-based
interventions
b) No
intervention
a) objective
measures of
smoking (saliva
thiocyanate
levels, alveolar
CO)
b) self-reported
smoking
behaviour
c) Intermediate
measures
(intentions,
attitude,
knowledge,
skills)
d) Media reach
a) RCT
b) Controlled
before and after
studies
The PICO(T) chart
18. Systematic review process
1. Well formulated question
2. Comprehensive data search
3. Unbiased selection and abstraction process
4. Critical appraisal of data
5. Synthesis of data
6. Interpretation of results
19. Searching for Information
• Types of
– Studies (RCTs, non-RCTs, cohort/case-controlled)
– Population and setting
– Interventions
– Outcome measures
• Cochrane Handbook and CRD Guidelines
– Both provide explanations regarding the different study designs, likely
biases and issues to consider when including them
www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf
http://handbook.cochrane.org/
20. WHERE TO SEARCH
Where to search
– Electronic databases: Medline, Embase, Cochrane,
PsycInfo, etc.
– Grey literature, dissertations, theses, conference
proceedings, national bodies (NICE, HTA), clinical trial
database (www.clincialtrails.gov/)
Librarians are your
friends!
21. SEARCH TERMS
• MESH terms and key words/synonyms
– Medical Subject Heading – controlled vocabulary
thesaurus used for indexing article
MeSH vocabulary is divided into types of terms:
- SUBJECT HEADINGS / DESCRIPTORS
- SUBHEADINGS / QUALIFIERS
- SUPPLEMENT CONCEPT RECORDS
25. Example
Mass media interventions to prevent smoking in
young people
P= Young people
Textwords:
Adolescent Girl
Child Boy
Juvenile Teenager
Young people Young adult
Student Youth
26. Example of search
P = YOUNG PEOPLE
MeSH Textwords
………………………. ……………………….
………………………. ……………………….
………………………. ……………………….
I = MASS MEDIA
MeSH Textwords
………………………. ……………………….
………………………. ……………………….
………………………. ……………………….
C = (if required)
O = PREVENTION OF SMOKING
MeSH Textwords
………………………. ……………………….
………………………. ……………………….
………………………. ……………………….
OR
OR
OR
P
AND
I
AND
C
AND
O
28. Selection of Studies
• Reference manager software package
– Endnote – RefMan – ProCite – Mendeley
• Import results and screen
– Assess titles/abstracts against your predetermined criteria
– If in doubt include
– Retrieve full text articles of initial selections
• Assess full text for inclusion
– Requires judgement (>1 reviewer)
– Check reviewer agreement (3rd review to resolve)
– Use a selection form to ensure consistency and record decisions
30. Critical appraisal
The process of systematically examining
research evidence to assess its validity,
results and relevance before using it to
inform a decision.
Alison Hill, Critical Appraisal Skills Programme, Institute of Health
Sciences, Oxford http://www.evidence-based-medicine.co.uk
31. Critical appraisal tools
“More than 35 different quality assessment
instruments have been published in the literature,
and most are designed for randomized clinical trials.”
• CASP
• PEDro Scale
• The JADAD Score
• GATE CAT
• SIGN
• CEBM- RCT
• JBI Checklist
• SURE
“The most common way to assess and report study
quality has been using a composite, numerical
scoring instrument.”
“The Jadad score and the T.C. Chalmers score are
two examples of quality assessment instruments.”
32. Assess the risk of bias and decide if
the results are trustworthy
33. • Validity •
• Internal validity: the extent to which the study is free from
bias
• Bias: systematic differences between groups
• Bias can be introduced because of the design, conduct, or
analysis of studies
• Low risk of bias: we can attribute differences in outcomes
to the differences in the treatment given and not other
variables (confounding)
34. External validity:
the extent to which the results apply outside the
study setting
– Can you use the results in your situation?
– Assess whether your patients/setting are
similar enough to those in the study
35.
36. Rapid risk of bias checklist
• Was the method of randomization truly random?
• Was allocation adequately concealed?
• Were groups comparable at the start?
• Were patients and practitioners providing care blinded?
• Was outcome assessment blinded or were outcomes
objective?
• Was there minimal loss to follow-up & losses explained?
• Was an intention-to-treat analysis conducted?
39. Narrative Synthesis Guidelines
• Developing a theory of how the intervention works, why and for whom
– Are the outcomes directly influenced by the intervention?
– Are there intermediate outcomes that should be considered when
systematically reviewing the evidence
• Developing a preliminary synthesis of the findings of included studies
– Describe each of the included studies
– Group the studies
– Tabulate results in order to identify patterns across the included studies
– Transform the data
– Translating data
http://cccrg.cochrane.org/author-resources
40. • Exploring relationships in the data within and between studies
– In order to explore reasons for differences in the direction
and size of effects of interventions
– Consider whether the following are likely to explain
differences between and within studies:
• Variability in outcomes;
• Variability in study designs;
• Variability in populations, interventions, settings;
• ‘Poor’ outcome measures used.
• Assessing the robustness of the synthesis
– The amount and quality of the evidence
– The methods used to synthesise the evidence
41. • Discussion
– Overall completeness and applicability of evidence
– Quality of evidence
– Potential biases of the review process
42. “Hang on. Systematic
reviews collect, appraise
and combine evidence.”
“So why do we need to
appraise them?”
Not all systematic reviews are high quality!
