Methods for adjusting survival estimates in the presence of treatment crossover: a simulation study

1. Methods for adjusting survival
estimates in the presence of
treatment crossover: a simulation
study
Nicholas Latimer, University of Sheffield
Collaborators: Paul Lambert, Keith Abrams, Michael Crowther, Allan
Wailoo and James Morden

2. 2
Contents
What is the treatment crossover problem
Potential solutions
Simulation study
Results
Conclusions

3. 3
Treatment switching – the problem
In RCTs often patients are allowed to switch from the control
treatment to the new intervention after a certain timepoint (eg
disease progression)
OS estimates will be confounded
OS is a key input into the QALY calculation
Cost effectiveness results will be inaccurate  an ITT analysis is
likely to underestimate the treatment benefit
Inconsistent and inappropriate treatment recommendations
could be made

4. 4
Treatment switching – the problem
Control Treatment
True OS difference
PFS PPS
Intervention
PFS PPS
Control  Intervention RCT OS
difference
PFS PPS
Survival time
Switching is likely to result in an underestimate of the treatment effect

5. What is usually done to adjust?
No clear consensus
Numerous „naive‟ approaches have been taken in NICE appraisals,
eg:
Very prone to
Take no action at all selection bias
Exclude or censor all patients who crossover – crossover
isn‟t random
Occasionally more complex statistical methods have been used, eg:
Rank Preserving Structural Failure Time Models (RPSFTM)
Inverse Probability of Censoring Weights (IPCW)
And others are available from the literature, eg:
Structural Nested Models (SNM)

6. 6
What are the consequences?
TA 215, Pazopanib for RCC [51% of control switched]
ITT: OS HR (vs IFN) = 1.26  ICER = Dominated
Censor patients: HR = 0.80  ICER = £71,648
Exclude patients: HR = 0.48  ICER = £26,293
IPCW: HR = 0.80  ICER = £72,274
RPSFTM: HR = 0.63  ICER = £38,925

7. Potential solutions
RPSFTM (and IPE algorithm)
Randomisation-based approach, estimates counterfactual survival times
Key assumption: common treatment effect
IPCW
Observational-based approach, censors xo patients, weights remaining patients
Key assumptions: “no unmeasured confounders”; must model OS and crossover
SNM
Observational version of RPSFTM
Key assumptions: “no unmeasured confounders”; must model OS and crossover

8. Simulation study (1)
None of these methods are perfect
But we need to know which are likely to produce least bias in different scenarios
Simulation study
Simulate survival data for two treatment groups, applying crossover that is linked to
patient characteristics/prognosis
In some scenarios simulate a treatment effect that changes over time
In some scenarios simulate a treatment effect that remains constant over time
Test different %s of crossover, and different treatment effect sizes
How does the bias and coverage associated with each method compare?

9. 9
Simulation study (2)
Methods assessed
Naive methods
ITT
Exclude crossover patients
Censor crossover patients
Treatment as a time-dependent covariate
Complex methods
RPSFTM
IPE algorithm
IPCW
SNM

10. 10
Results: common treatment effect
RPSFTM / IPE worked very well
IPCW and SNM performed ok when crossover % was lower
50.00 ITT IPCW RPSFTM IPE SNM
40.00
AUC mean bias (%)
30.00
20.00
10.00
0.00
19 20 27 28 31 32 39 40 43 44 51
-10.00
IPCW and SNM performed poorly when crossover % was very high
Naive methods performed poorly (generally led to higher bias than ITT)

11. 11
Results: effect 15% in xo patients
RPSFTM / IPE produced higher bias than previous scenarios
IPCW and SNM performed similarly to RPSFTM / IPE providing crossover < 90%
ITT IPCW RPSFTM IPE SNM
45.00
AUC mean bias (%)
35.00
25.00
15.00
5.00
-5.00
17 18 25 26 29 30 37 38 41 42 49
-15.00
IPCW and SNM performed poorly when crossover % was very high
Bias not always lower than that associated with the ITT analysis

12. 12
Results: effect 25% in xo patients
RPSFTM / IPE produced even more bias
IPCW and SNM produce less bias than RPSFTM / IPE providing crossover < 90%
50.00 ITT IPCW RPSFTM IPE SNM
AUC mean bias (%)
40.00
30.00
20.00
10.00
0.00
23 24 33 34 35 36 45 46 47 48 57
-10.00
-20.00
No „good‟ options when crossover % is very high
Often ITT analysis likely to result in least bias (esp. when trt effect low)

13. 13
Conclusions
Treatment crossover is an important issue that has come to the fore in HE arena
Current methods for dealing with treatment crossover are imperfect
Our study offers evidence on bias in different scenarios (subject to limitations)
RPSFTM / IPE produce low bias when treatment effect is common
 But are very sensitive to this
IPCW / SNM are not affected by changes in treatment effect between groups, but in
(relatively) small trial datasets observational methods are volatile
 Especially when crossover % is very high (leaving low n in control group)
Very important to assess trial data, crossover mechanism, treatment effect to
determine which method likely to be most appropriate
 Don’t just pick one!!

14. 14
Back-up Slides
28/09/2012 © The University of Sheffield

15. 15
Data Generation (1)
Used a two-stage Weibull model to generate underlying survival times
and a time-dependent covariate (called „CEA‟)
Longitudinal model for CEA (for ith patient at time t):
where
is the random intercept
is the slope for a patient in the control arm
is the slope for a patient in the treatment arm (all
is the change in the intercept for a patient with bad prognosis
compared to a patient without bad prognosis
Picked parameter values such that CEA increased over time, more
slowly in the experimental group, and was higher in the badprog group
28/09/2012 © The University of Sheffield

16. 16
Data Generation (2)
The survival hazard function was based upon a Weibull (see Bender
et al 2005):  1
h(t )   t exp( X )
In our case,
X = 1 * trt i + ( * log(t) ) * trt i +  2 badprogi +  (cea(t) )
where  1 is the log hazard ratio (the baseline treatment effect)
 is the time-dependent change in the treatment effect
 2 is the impact of a bad prognosis baseline covariate on
survival
 is the coefficient of CEA, indicating its effect on survival
We used this to generate our survival times
 So, CEA has an effect on survival over time, and there is a separate
time-dependent treatment effect
28/09/2012 © The University of Sheffield

17. 17
Data Generation (3)
We applied a reduced treatment effect to crossover patients – this was
equivalent to the baseline treatment effect multiplied by a factor to
ensure that this effect was less than (or equal to) the average effect
received in the experimental group
28/09/2012 © The University of Sheffield

18. 18
Data Generation (4)
We then selected parameter values in order that „realistic‟ datasets
were created:
1.00
2.5
AF actual
0.80
AF predicted
trtrand = 0 trtrand = 1
2
0.60
Acceleration Factor
1.5
0.40
0.20
1
0.00
0.5
0 200 400 600 800 1000 1200
analysis time
Number at risk
0
trtrand = 0 251 154 86 52 25 9 0
0 200 400 600 800 1000 1200
trtrand = 1 249 186 126 71 42 21 0
Time (days)
28/09/2012 © The University of Sheffield

19. 19
Data Generation (5)
We made several assumptions about the „crossover mechanism‟:
1. Crossover could only occur after disease progression (disease
progression was approximately half of OS, calculated for each patient
using a beta(5,5) distribution)
2. Crossover could only occur at 3 „consultations‟ following disease
progression
These were set at 21 day intervals
Probability of crossover highest at initial consultation, then falls in second
and third
3. Crossover probability depended on time-dependent covariates:
CEA value at progression (high value reduced chance of crossover)
Time to disease progression (high value increased chance of crossover)
This was altered in scenarios to test a simpler mechanism where probability
only depended on CEA
 Given all this, CEA was a time-dependent confounder
28/09/2012 © The University of Sheffield

20. 20
Scenarios
Variable Value Alternative
Sample size 500 
Number of prognosis groups (prog) 2 
Probability of good prognosis 0.5 
Probability of poor prognosis 0.5 
Maximum follow-up time 3 years (1095 days) 
Multiplication of OS survival time due Log hazard ratio = 0.5 
to bad prognosis group
Survival time distribution Alter parameters to test two levels of disease 
severity
Initially assumed treatment effect Alter to test two levels of treatment effect 
Time-dependence of treatment effect Treatment effect received is equal in all crossover 
patients, and equals baseline treatment effect
multplied by a factor. However set α to zero in
some scenarios. Also include additional treatment
effect decrement in crossover patients in some
scenarios
Probability of switching treatment Test two levels of treatment crossover proportions 
over time
Prognosis of crossover patients Test three crossover mechanisms in which different 
groups become more likely to cross over
28/09/2012 © The University of Sheffield

21. 21
Scenarios
Variable Value Alternative
Sample size 500 
Number of prognosis groups (prog) 2 
Probability of good prognosis 0.5 
Probability of poor prognosis 0.5 
Maximum follow-up time 3 years (1095 days) 
Multiplication of OS survival time due Log hazard ratio = 0.5 
to bad prognosis group
Survival time distribution Alter parameters to test two levels of disease  (2)
severity
Initially assumed treatment effect Alter to test two levels of treatment effect 
Time-dependence of treatment effect Treatment effect received is equal in all crossover 
patients, and equals baseline treatment effect
multplied by a factor. However set α to zero in
some scenarios. Also include additional treatment
effect decrement in crossover patients in some
scenarios
Probability of switching treatment Test two levels of treatment crossover proportions 
over time
Prognosis of crossover patients Test three crossover mechanisms in which different 
groups become more likely to cross over
28/09/2012 © The University of Sheffield

22. 22
Scenarios
Variable Value Alternative
Sample size 500 
Number of prognosis groups (prog) 2 
Probability of good prognosis 0.5 
Probability of poor prognosis 0.5 
Maximum follow-up time 3 years (1095 days) 
Multiplication of OS survival time due Log hazard ratio = 0.5 
to bad prognosis group
Survival time distribution Alter parameters to test two levels of disease  (2)
severity
Initially assumed treatment effect Alter to test two levels of treatment effect  (4)
Time-dependence of treatment effect Treatment effect received is equal in all crossover 
patients, and equals baseline treatment effect
multplied by a factor. However set α to zero in
some scenarios. Also include additional treatment
effect decrement in crossover patients in some
scenarios
Probability of switching treatment Test two levels of treatment crossover proportions 
over time
Prognosis of crossover patients Test three crossover mechanisms in which different 
groups become more likely to cross over
28/09/2012 © The University of Sheffield

23. 23
Scenarios
Variable Value Alternative
Sample size 500 
Number of prognosis groups (prog) 2 
Probability of good prognosis 0.5 
Probability of poor prognosis 0.5 
Maximum follow-up time 3 years (1095 days) 
Multiplication of OS survival time due Log hazard ratio = 0.5 
to bad prognosis group
Survival time distribution Alter parameters to test two levels of disease  (2)
severity
Initially assumed treatment effect Alter to test two levels of treatment effect  (4)
Time-dependence of treatment effect Treatment effect received is equal in all crossover  (8)
patients, and equals baseline treatment effect
multplied by a factor. However set α to zero in (12)
some scenarios. Also include additional treatment
effect decrement in crossover patients in some
scenarios
Probability of switching treatment Test two levels of treatment crossover proportions 
over time
Prognosis of crossover patients Test three crossover mechanisms in which different 
groups become more likely to cross over
28/09/2012 © The University of Sheffield

24. 24
Scenarios
Variable Value Alternative
Sample size 500 
Number of prognosis groups (prog) 2 
Probability of good prognosis 0.5 
Probability of poor prognosis 0.5 
Maximum follow-up time 3 years (1095 days) 
Multiplication of OS survival time due Log hazard ratio = 0.5 
to bad prognosis group
Survival time distribution Alter parameters to test two levels of disease  (2)
severity
Initially assumed treatment effect Alter to test two levels of treatment effect  (4)
Time-dependence of treatment effect Treatment effect received is equal in all crossover  (8)
patients, and equals baseline treatment effect
multplied by a factor. However set α to zero in (12)
some scenarios. Also include additional treatment
effect decrement in crossover patients in some
scenarios
Probability of switching treatment Test two levels of treatment crossover proportions  (24)
over time
Prognosis of crossover patients Test three crossover mechanisms in which different 
groups become more likely to cross over
28/09/2012 © The University of Sheffield

25. 25
Scenarios
Variable Value Alternative
Sample size 500 
Number of prognosis groups (prog) 2 
Probability of good prognosis 0.5 
Probability of poor prognosis 0.5 
Maximum follow-up time 3 years (1095 days) 
Multiplication of OS survival time due Log hazard ratio = 0.5 
to bad prognosis group
Survival time distribution Alter parameters to test two levels of disease  (2)
severity
Initially assumed treatment effect Alter to test two levels of treatment effect  (4)
Time-dependence of treatment effect Treatment effect received is equal in all crossover  (8)
patients, and equals baseline treatment effect
multplied by a factor. However set α to zero in (12)
some scenarios. Also include additional treatment
effect decrement in crossover patients in some
scenarios
Probability of switching treatment Test two levels of treatment crossover proportions  (24)
over time
Prognosis of crossover patients Test three crossover mechanisms in which different  (72)
groups become more likely to cross over
This combined to 72 scenarios
28/09/2012 © The University of Sheffield

26. 26
Scenarios
Variable Value Alternative
Sample size 500 
Number of prognosis groups (prog) 2 
Probability of good prognosis 0.5 
Probability of poor prognosis 0.5 
Maximum follow-up time 3 years (1095 days) 
Multiplication of OS survival time due Log hazard ratio = 0.5 
to bad prognosis group
Survival time distribution Alter parameters to test two levels of disease 
severity
Initially assumed treatment effect Alter to test two levels of treatment effect 
Time-dependence of treatment effect Treatment effect received is equal in all crossover 
patients, and equals baseline treatment effect
multplied by a factor. However set α to zero in
some scenarios. Also include additional treatment
effect decrement in crossover patients in some
scenarios
Probability of switching treatment Test two levels of treatment crossover proportions 
over time
Prognosis of crossover patients Test three crossover mechanisms in which different 
groups become more likely to cross over
28/09/2012 © The University of Sheffield

27. 27
Estimating AUC
1. ‘Survivor function’ approach
Apply treatment effect to survivor function (or hazard function)
estimated for experimental group  calculate AUC
2. ‘Extrapolation’ approach
Extrapolate counterfactual dataset to required time-point (only
relevant for RPSFTM/IPE approaches)  calculate AUC
3. ‘Shrinkage’ approach
Use estimated acceleration factor to „shrink‟ survival times in
crossover patients in order to obtain an adjusted dataset  calculate
AUC (only relevent for AF-based approaches)
28/09/2012 © The University of Sheffield

28. A topical analogy... 28
England Spain
Vs
(control group) (intervention group)
Kick-off...

29. A topical analogy... 29
Halftime: England 0 – 3 Spain
 Spain are statistically significantly
better than England
 For ethical reasons, the
Spanish team are cloned and the
English team are sent home. So
the second half is Spain Vs
Spain

30. A topical analogy... 30
Full time: England 2 – 5 Spain
 Spain still win, but not as comfortably as
they would have
Switching is likely to result in an underestimate of
the true supremacy of the intervention

31. 31
Results (4)
While the SNM and IPCW methods appeared to produce similar levels of bias, the
SNM approach was more volatile:
Relatively often failed to converge (in up to 90% of sims in one scenario)
Typically when disease severity was high and treatment effect was low
Only produced lower bias than the ITT analysis in 38% of scenarios
(compared to 60% for the IPCW method)
Was more sensitive to increasing crossover %
28/09/2012 © The University of Sheffield

32. 32
Results (5)
Relationship between bias and treatment crossover %
140
120
100
80
Mean % bias
60
40
ITT
20
0
60% 70% 80% 90% 100%
-20
-40
Crossover proportion (at-risk patients)
28/09/2012 © The University of Sheffield

33. 33
Results (5)
Relationship between bias and treatment crossover %
140
120
100
80
Mean % bias
60 IPE
RPSFTM
40
ITT
20
0
60% 70% 80% 90% 100%
-20
-40
28/09/2012 © The University ofCrossover
Sheffield proportion (at-risk patients)

34. 34
Results (5)
Relationship between bias and treatment crossover %
140
120
100
80
Mean % bias
IPCW
60
IPE
40 RPSFTM
ITT
20
0
60% 70% 80% 90% 100%
-20
-40
28/09/2012 © The University ofCrossover
Sheffield proportion (at-risk patients)

35. 35
Results (5)
Relationship between bias and treatment crossover %
140
120
100
80
IPCW
Mean % bias
60 IPE
RPSFTM
40
SNM
20 ITT
0
60% 70% 80% 90% 100%
-20
-40
28/09/2012 © The University ofCrossover
Sheffield proportion (at-risk patients)

36. 36
Conclusions
Treatment crossover is an important issue that has come to the fore in HE arena
Current methods for dealing with treatment crossover are imperfect
Our study offers evidence on bias in different scenarios (subject to limitations)
1. RPSFTM / IPE produce low bias when treatment effect is common
2. When treatment effect ≈ 15% lower in crossover patients RPSFTM / IPE and IPCW /
SNM methods produce similar levels of bias (5-10%)
[provided suitable data available for obs methods and <90% crossover]
3. When treatment effect ≈ 25% lower IPCW/SNM produce less bias than RPSFTM/IPE
[provided suitable data available for obs methods and <90% crossover]
[and significant bias likely to remain  ITT analysis may offer least bias]
4.Very important to assess trial data, crossover mechanism, treatment effect to
determine which method likely to be most appropriate
 Don’t just pick one!!