1. Superiority-Equivalence-Non-Inferiority Trials:
What does it all mean?
Gerald Gartlehner

2. What decision-makers want to know
Is the new treatment
better than the
established one ?
If not, is it equally
effective and preferable
for some other reason
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

3. Overview
 Definitions and concepts
 Challenges of non-inferiority trials
 What does this mean for systematic reviews?
 Language considerations for comparative
effectiveness reviews
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

4. Proving Efficacy
1. Showing superiority of one
(new) treatment over
another (placebo or active)
2. Showing equivalence or non-
inferiority of a new
intervention relative to an
already existing efficacious
treatment
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

5. The interpretation of non-inferiority and
equivalence can be confusing
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

6. Ranking position: 72
Ranking position: 34

7. Gerald Gartlehner
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

8. Superiority game: the winner takes it all….
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

9. Equivalence game – equally good or clearly
better
 Equivalence margin: tied or less than 1 goal
difference
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

10. Non-inferiority game– at least not substantially
worse….
 Non-inferiority margin: can‘t lose with more
than 1 goal difference
Gerald Gartlehner
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

11. Definitions
Superiority trial
Objective: To determine a clinically relevant difference
between two interventions
Equivalence trial
Objective: To determine whether a (new) intervention is
neither worse nor better than another (established)
intervention
Non-inferiority trial
Objective: To determine whether a (new) intervention is not
inferior to another (established) intervention
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

12. Equivalence - Non-Inferiority
The naïve approach:
 If a head-to-head trial shows no statistically significant
difference, two interventions are “equivalent”
 Problem – underpowered studies or high variance will
create “equivalent” treatments
The statistical approach:
 Define a margin of non-inferiority or equivalence
 If 95% confidence interval of the difference DOES NOT
cross the margin, the new intervention is non-inferior or
equivalent
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

13. Arthroscopy vs. sham arthoscopy in patients
with knee osteoarthritis
favors sham favors arthroscopy
MID
20 10 0 10 20
Knee Specific Pain Scale: difference in points
Mosley et al. New England J Med, 2002;347:81-88
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

14. Non-inferiority margin (d)
The limit of acceptable inferiority:
Minimal important difference
Clinical judgement
Statistical considerations
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

15. Determining the non-inferiority margin
statistically
Standard vs. placebo
2 points (1.5 to 2.5)
New vs. standard
2 1 0
Maximimum non-inferiority margin: dmax = 1.5
Fractional preservation of treatment effect: f = 0.5
d = dmax x (1 – f)
d = 1.5 x ( 1 – 0.5) = 0.75

16. Peculiar issues of non-inferiority trials: the
backwardness
Null hypothesis and alternative hypothesis are
reversed
Type I and type II errors are reversed
Per-protocol analyses can be more important than
ITT analysis (ITT analyses are biased towards finding
no difference)
P-value is one-sided (0.025)

17. Assay Sensitivity and Constancy Assumption
The ability of a trial to distinguish effective from
ineffective treatments (depends on the effect size
the trial wants to detect).
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

18. Assay Sensitivity
Assumptions in non-inferiority trials:
 The efficacy of the active control was preserved in
the non-inferiority study (i.e. that it had assay
sensitivity).
 If it was not, equivalence or non-inferiority
conclusions are meaningless (The non-inferior drug
could have no effect at all).

19. Constancy Assumption
 Active comparator must be well established and have
predictable and consistent treatment effects
 Participants must be similar to those in trial
establishing efficacy
 Outcomes must be similar to those in trials
establishing efficacy

20. Biocreep
Effective treatment
Still clinically relevant?
2 1 0
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

21. Key Points for critical appraisal
Was a non-inferiority margin defined based on
clinical considerations and statistical reasoning?
Was it established a priori?
Was the study powered based on the non-inferiority
margin?
Was an ITT and a per-protocol analysis conducted?
Was the trial design (e.g. eligibility criteria)
consistent with placebo controlled trials of the
established treatment?

22. What does this mean for systematic reviews ?
For meta-analyses – data can be used just as
from any superiority trial
For qualitative assessments – language
considerations
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

23. Language considerations
“..there was no statistically significant difference
between A and B..”
“..studies failed to show a difference..”
Can mean:
1) The evidence shows equivalence
2) The evidence is inconclusive (because confidence
intervals are wide-lack of precision)
AHRQ guidance: Assessing equivalence and non-inferiority [draft]
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

24. Language considerations
Better:
“..treatments A and B had similar mortality rates..”
Expressing non-inferiority:
“…treatment A is at least as effective as treatment B for
[Outcome or study objective]…”
AHRQ guidance: Assessing equivalence and non-inferiority [draft]
Österreichische Cochrane Zweigstelle (ÖCZ) ∙ www.cochrane.at Gerald Gartlehner
Department für Evidenzbasierte Medizin und Klinische Epidemiologie, Donau-Universität Krems

25. Do we need to establish equivalence
margins for CERs ?