The document provides an overview of the NICE appraisal process to help manufacturers improve their submissions and successfully achieve reimbursement. It discusses key concepts like opportunity cost and budget impact. It highlights common errors in submissions around modeling assumptions and mixed treatment comparisons. It also covers issues like crossover trials, assessing innovation, patient access schemes, and considerations around value-based pricing. The overall aim is to empower manufacturers with insights into the appraisal process to make well-informed decisions.

1. “Breaking Down the Appraisal
Process to Empower Manufacturers
to Improve Internal Decision-
Making and Successfully Achieve
Reimbursement”
Matt Stevenson

2. Quick Intro
• NICE Appraisal Committee Member
• Working Group Member for the revision of the
NICE Methods Guide
• Technical Director for ScHARR-TAG (the largest
academic group undertaking work for NICE)
• Professor of Health Technology Assessment

3. Outline
• The concept of opportunity cost
• Errors seen in submissions
• Mixed Treatment Comparisons
• Crossover in RCTs
• Innovation
• Patient Access Schemes / Rapid Reviews
• Value Based Pricing

4. Opportunity Cost
• Should be a familiar concept to all, as all decisions on
food, refreshment, housing, schooling, leisure, holidays,
savings ..... are taken in the context of a fixed budget
• Any increased expenditure on one item will result in a
reduction in expenditure of another item.
• Given a fixed healthcare budget, the uptake of drug X will
mean a reduction in the use of drug Y.

5. Theory behind the threshold
approach
• In principle, the adoption of any new technology, with a
cost burden, would result in the least cost-effective
option currently used being displaced (regardless of
disease area)
• Assuming zero inflation this is a downward force on the
threshold.
• This process aims to increase societal QALYs . The
converse view to ‘drug rationing’ is ‘health maximisation’

6. Limitations
• The process fails if PCTs reduce expenditure in cost-
effective procedures rather than those that are not cost-
effective
• Is the current threshold correct?

7. Alternative approaches?
“I'm afraid we've got a
budget problem, Marge.
Your boy picked a bad
time to fall down a well.
If he had done it at the
beginning of the fiscal
year, no problemo.”

8. Errors seen in submissions
Some errors are inexcusable:
• Incorporating the unconfirmed price of the drug,
or the discount rate in the probabilistic analyses
• Assuming different utilities for patients prior to
receiving the intervention and comparator
• ‘Broken’ models
• PAS altering results for comparator drugs

9. Omission of correlation
Sampling from a simple linear regression
20 20
15 X X 15 X X
X X X X
10
XX 10
XX
X X
X X X X
5
X X 5 X
X X X X X X X
-1 0 X 1 2 x 3 4 5 -1 0 X 1 2 x 3 4 5
Slope and intercept Slope and intercept
sampled sampled dependently
independently

10. Ignoring extended dominance
As the gradient from A to B is greater
10000
than the gradient from A to C and C 9000
B
C
produces more QALYs than B then B 8000
7000
suffers extended dominance by A and 6000
Cost (£)
5000
C. By using a combination of A and C 4000
A
3000
the same QALYs can be produced as 2000
in B for a lower cost, or alternatively 1000
0
0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9
for the same expenditure as in B, QALYs Gained
more QALYs can be accrued.
If ICERs do not monotonically increase then interventions
which are extendedly dominated have not be removed.

11. Misinterpretation of CEACs
The CEAC shows the
probability of being most
cost-effective, but does
not incorporate the
ramifications of being
non-optimal, nor indicate
that similarity in the
results of strategies may
have similar.
The mean cost per QALY
must be presented.

12. Misinterpretation of CEACs
To reinforce the point. Being paid £1
to play Russian Roulette (6
chambers, 1 bullet), would produce
the following CEAC

13. Choosing when to extrapolate
Base case ICER ICER - Based on alternative time points In this example the
£100,000
ICER
manufacturer defined
£90,000 an arbitrary rule as to
when a parametric
£80,000
curve should replace
£70,000
the Kaplan Meier data.
In this example the
£60,000
assumed switch
£50,000
Cycles appeared favourable to
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
the intervention.

14. Refusal to look at sequences
• Often treatments should be aimed at a second-line
position, due to a cheap, safe and effective
incumbent treatment.
• In one appraisal a company refused to consider
sequencing claiming it was unethical given their
model indicated the intervention was more cost-
effective than the comparator.
• This was a misunderstanding of the need for a full
incremental analysis

15. Inappropriate assumption of
certainty
• In the appeal of sorafenib for HCC a key issue
centred on whether the survival distribution was
better represented by: a lognormal distribution
(≈£51,000 per QALY); a weibull distribution (CIC
but considerable higher); or whether a prudent
view would be to assume it lay somewhere
inbetween.

16. Continually selecting assumptions
that favour the intervention.
• This is akin to ‘resurrection
with a thousand bandages’.
• The committee and the
Assessment Groups are
experienced and are very
likely to see that the central
ICER is biased.

17. Mixed Treatment Comparisons
• MTCs (also known as Network Meta Analyses) are
becoming more common place in assessing the efficacy
of treatments. There has been a rapid rise in the number
of MTCs being presented to NICE appraisal committees.
• Although NICE maintains a preference for Head to Head
studies, the evidence from MTCs are fully considered.
When multiple treatments are considered, the
distinction between ‘direct’ and ‘indirect’ analyses is
difficult to determine.

18. A Taxonomy of Comparisons
A B Direct Comparison (Head to head)
A
B
‘Naïve’ Indirect Comparison:
Absolute effect estimates from individual trial arms
A
C ‘Adjusted’ Indirect Comparison:
B C Relative effect estimates between treatments
A B Mixed Treatment Comparison/’Network’
C Meta-Analysis:
A ‘Adjusted’ indirect comparison extended to
B C more complex networks of trial evidence (i.e.
head to head and indirect evidence)

19. Are ‘Adjusted’ Indirect
Comparisons Systematically
Biased?
Song et al (2003). BMJ, 326:472-475

20. • Significant discrepancy in 3 out of 44 comparisons (95%
CI of difference between methods did not cross zero)
• Equally likely to over or underestimate difference
• Indirect comparisons may be useful when direct evidence
is absent or insufficient. Indirect evidence could
strengthen conclusions based on direct evidence
• Direct evidence generally regarded as best, but at times
may be flawed

21. One compulsion for
performing MTCs
“. . . to ignore indirect evidence either makes
the unwarranted claim that it is irrelevant,
or breaks the established precept of
systematic review that synthesis should
embrace all available evidence”
Lu & Ades (2004). Combination of direct and indirect evidence in
mixed treatment comparisons. Stat Med, 23: 3105-3124

22. Recommendations – Song (2009)
• Literature search needs to be systematic in order to
identify all relevant studies
• Naïve indirect approaches should be avoided
• Methods for investigating heterogeneity in standard meta-
analysis can be applied to assess trial similarity in indirect
comparisons (e.g. subgroups, meta-regression etc)

23. Recommendations – Song (2009)
• Evidence from head to head RCTs should not be excluded
in analyses that use indirect comparisons
• Direct and indirect evidence should be separately
presented and compared
• Consistency should be explicitly assessed before direct and
indirect evidence is combined

24. Crossover within RCTs
• Where an intervention has been shown to be efficacious
it may be unethical to leave patients on the comparator
drug. This confounds the standard intention to treat
analysis.
• Two methods have been used to adjust for the crossover
in submissions to NICE: the Rank Preserving Structural
Failure Time model (RPSFT); and the Inverse Probability
of Censoring Weights (IPCW)

25. RPSFT
• The RPSFT uses the randomisation assumption
to estimate treatment effect such that
counterfactual survival (a function of observed
survival time, observed treatment and the
treatment effect) would have been equal in
randomised groups, had no experimental
treatment been given to any patients.

26. IPCW
• The IPCW approach treats crossover patients as
informatively censored, and applies time-varying
weights to uncensored survival probabilities
based on the probability of patients crossing
over given their covariate history. The IPCW
approach is reliant on the assumption of no
unmeasured confounders, and upon the
existence of a reasonable number of uncensored
observations.

27. Crossover within RCTs
• Research is currently ongoing to
determine which method is most
applicable in health technology evaluations

28. Innovation
• NICE is expected to take into account the potential
for long-term benefits to the NHS of innovation
• Above a most plausible ICER of £20,000 the
Committee will consider the innovative nature of
the technology, specifically if the innovation adds
demonstrable and distinctive benefits of a
substantial nature which may not have been
adequately captured in the QALY measure.

29. Innovation
• However, should patient health be sacrificed for
the pursuit of innovation, resulting in a higher
threshold for innovative products?
• Industry may have a conflicting view to those
tasked with maximising societal health from the
fixed budget.

30. Where cost/QALY at threshold
£200,000,000
Patent expires and generic entry at year 15 Value of the innovation
£180,000,000 Generic price are 25% of the brand
Discount rate of 3.5%
£160,000,000
Private sector share
resent value of innovation at T
£140,000,000
£120,000,000
£100,000,000
£80,000,000
£60,000,000 NHS share
P
£40,000,000
£20,000,000
£0
0 5 10 15 20 25 30
Years from launch (T)

31. Enhanced price for ‘Innovation’
£300,000,000
Accept price > P* during patent
Private sector share
because price < P* when generics enter
£250,000,000
£200,000,000
Value of the innovation
resent value of innovation at T
£150,000,000
£100,000,000
£50,000,000
£0
P
0 5 10 15 20 25 30 35 40
NHS share
-£50,000,000
-£100,000,000
-£150,000,000
Years from Launch (T)

32. Patient Access Schemes
• It was perceived that the manufacturer would
incorporate a PAS with the initial submission, or
following the release of guidance (using a Rapid Review)
• In exceptional circumstances(!) a PAS would be
accepted following the ACD.

33. Historic use of PAS
• Up to June 2012, 29 products have submitted PAS.
• The initial PAS had been criticised for imposing a burden
both in terms of administrative and operational costs
(Williamson, Lancet 2010). As time has progressed there
has been a move towards commercial in confidence
discounts within PAS, which are viewed as less
burdensome

34. Recommendations
following a PAS
• The trend in the acceptance of products with PAS is
interesting.
1
0
-1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

35. Comments on PAS
• The ability for the manufacturer to submit multiple PAS
(albeit it requiring the agreement with DH) allows the
potential for some gaming
• Recommending an intervention at the threshold will not
produce an increase in societal QALYs.
• Can be viewed as a fore-runner to value based pricing,
with NICE moving towards a position of price setter.

36. Value Based Pricing
However..........

37. Some key principles
• Any additional weight provided to the young will
result in a lesser weight being provided to the
elderly and vice versa. Ditto for male/female
severe condition / mild condition .......
• Any additional benefits provided to evaluated
interventions must also be taken into
consideration for those items that will be
displaced.

38. Some key principles
• If some interventions are ‘recommended’
following VBP that wouldn’t have been
under the current system, it follows that
there will be interventions that would have
been ‘recommended’ that are now not.
• This is, in essence, a zero-sum game

39. Additional details
• It is likely that NICE will no longer have the powers to
recommend (or not) interventions. It will instead be the
body that decides on the value offered by an
intervention.
• Setting a value based price is not compulsory. The only
sanction is that “it would be the company’s responsibility
to explain to the public why it was not prepared to offer
that drug at an appropriate price”

40. Any Questions?