How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies
•Transferir como PPTX, PDF•
0 gostou•254 visualizações
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Semelhante a How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies
Semelhante a How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies (20)
Último
Último (20)
How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies
- 1. HOW TO BUILD A MODEL TO PREDICT REIMBURSEMENT DECISIONS BY MAJOR HTA/CER AGENCIES Kermit Daniel, PhD – Chief Analytics Officer, Context Matters, Inc. December 2013 Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED
- 2. BUILDING A PREDICTIVE MODEL OF HTA BEHAVIOR • Why build a model? • Can we build a model? • How do we build a model? • How will we know if we built a good model? Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 2
- 3. WHY BUILD A MODEL? A GOOD MODEL WILL • Improve predictions • Disentangle multiple influences • Provide actionable guidance for how to increase the likelihood of a positive reimbursement decision Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 3
- 4. WHY BUILD A MODEL? A MODEL IS • Useful simplification • Mathematical: Y = Xβ + ε Y: Probability of a positive recommendation X: Observable influences of recommendations β: Parameters we will estimate ε: Effect of influences we don’t observe Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 4
- 5. CAN WE BUILD A MODEL? WHAT HAS TO BE TRUE? • Agencies behave in ways that can be described by a simple model that we can estimate o Consistency - non-random, consistent behavior • Through time • Across therapeutic areas • Across agencies o Transparency – we can observe decision factors (or good proxies) Y = Xβ + ε THIS EXPLAINS A LOT THIS EXPLAINS A LITTLE Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 5
- 6. CONSISTENCY ACROSS AGENCIES AGENCIES FREQUENTLY AGREE WHEN PRESENTED WITH THE SAME DRUG/INDICATION TO REVIEW FREQUENCY OF AGREEMENT BETWEEN AGENCY PAIRS CADTH HAS NICE PBAC SMC 68% 78% 85% 76% PBAC 63% 72% 85% NICE 61% 65% HAS 58% Note: Reviews of 94 drugs reviewed by at least two agencies between January 2005 – February 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 6
- 7. CONSISTENCY BETWEEN DECISION AND ASSESSMENTS THE CLINICAL AND/OR ECONOMIC ASSESSMENT AND THE REIMBURSEMENT DECISION ARE USUALLY CONSISTENT AGENCY PAIR Same Clinical Outcome Different Clinical Assessment PBAC:SMC P value Same Economic Outcome Different Economic Assessment 0.05* P value 0.00* Agree on decision 27 4 20 7 Disagree on decision 5 4 1 8 CADTH:PBAC 0.00* 0.14 Agree on decision 16 4 11 8 Disagree on decision 2 11 3 9 SMC:HAS Agree on decision 0.01* 32 9 4 9 Disagree on decision Note: Reviews of 94 drugs reviewed by at least two agencies between January 2005 – February 2013. Asterisk and shading indicates significance at 5% or better. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 7
- 8. TRANSPARENCY WE OBSERVE A LOT ABOUT REVIEWS, INCLUDING . . . DRUGS CLINICAL STUDIES SUBMISSIONS • Chemical type • Indications • Regulatory history • Orphan status • Number, dates • Design, e.g., goal, comparators, size, subpopulations, length • HTA assessment • Number • Professional, patient, other group support AGENCIES ECONOMIC MODELS CONCLUSIONS • Identity • Region • Agency-specific factors, e.g., additional benefit • Manufacturer comparators & assumptions • Agency model • HTA assessment • Recommendation • Restrictions • Factors discussed, e.g., study limitations, PRO use Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 8
- 9. HOW DO WE BUILD A MODEL? HOW DO WE DECIDE WHAT DATA TO INCLUDE? • Predict outcomes that matter • Capture implications of agency objectives and constraints • Incorporate what we observe about agency behavior – but this is dangerous! Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 9
- 10. PREDICT OUTCOMES THAT MATTER POSITIVE DECISIONS OFTEN ADD RESTRICTIONS Oncology NICE Non-Oncology 55% 54% 46% 45% Positive Decision Not More Restrictive Positive Decision More Restrictive Note: Based on 150 NICE reviews of 72 drugs for 34 diseases between January 2007 – August 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 10
- 11. CAPTURE IMPLICATIONS OF WHAT WE OBSERVE ABOUT AGENCY BEHAVIOR PRIMARY OUTCOMES THAT ARE PROs ARE MORE LIKELY TO BE MENTIONED IN THE DECISION RATIONALE Neurology and Respiratory Indications Note: Based on a Chi-squared test, the difference between the observed frequencies and the expected frequencies were statistically significant at the .01 level. Data span 2005 – April 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 11
- 12. THE DANGER OF A “DATA MINING” APPROACH RANDOM VARIABLES Correlation with Y X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 -0.35 -0.30 0.28 0.17 -0.23 0.20 -0.14 -0.01 0.07 0.22 -0.21 -0.07 -0.33 0.14 0.03 -0.02 0.28 -0.19 0.12 X20 X21 X22 X23 X24 X25 X26 X27 X28 X29 X30 X31 X32 X33 X34 X35 X36 X37 X38 0.09 0.21 -0.02 -0.22 0.19 0.38 0.15 -0.05 0.15 -0.39 -0.24 0.30 0.00 -0.20 -0.19 0.12 0.13 -0.27 -0.12 X39 X40 X41 X42 X43 X44 X45 X46 X47 X48 X49 X50 X51 X52 X53 X54 X55 X56 X57 -0.26 0.38 0.04 -0.41 -0.25 -0.32 -0.27 0.06 0.21 0.34 0.02 0.21 -0.36 0.22 -0.12 0.06 0.13 0.12 0.26 X58 X59 X60 X61 X62 X63 X64 X65 X66 X67 X68 X69 X70 X71 X72 X73 X74 X75 X76 Possible Drivers -0.31 -0.03 0.01 -0.23 -0.12 -0.27 0.21 0.03 0.10 -0.22 -0.36 -0.17 0.08 0.15 -0.52 -0.01 0.05 -0.04 0.10 X77 X78 X79 X80 X81 X82 X83 X84 X85 X86 X87 X88 X89 X90 X91 X92 X93 X94 -0.22 -0.03 -0.11 0.08 0.07 -0.20 0.01 -0.21 -0.23 -0.27 0.06 0.04 0.30 0.06 -0.20 0.09 0.11 0.42 X25 X29 X40 X42 X51 X68 X72 X94 Shading indicates correlation with Y is significant at 5%. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 12
- 13. THE DANGER OF A “DATA MINING” APPROACH WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL • All estimates significant at better than 5% • Two are significant at .1% or better reg Y X72 X42 X25 X40 • Very unlikely to have occurred by chance (p=1/10,000) • Explains 2/3 of the variation in Y X51; Source | SS df MS -------------+-----------------------------Model | 1684.41859 5 336.883717 Residual | 909.007507 24 37.8753128 -------------+-----------------------------Total | 2593.42609 29 89.4284859 Number of obs F( 5, 24) Prob > F R-squared Adj R-squared Root MSE = = = = = = 30 8.89 0.0001 0.6495 0.5765 6.1543 -----------------------------------------------------------------------------Y | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------X72 | -1.654714 .5923167 -2.79 0.010 -2.877196 -.4322327 X42 | -1.369594 .5741604 -2.39 0.025 -2.554603 -.1845855 X25 | 1.435235 .6360141 2.26 0.033 .1225669 2.747904 X40 | 1.329745 .5173819 2.57 0.017 .2619215 2.397569 X51 | -1.138341 .5299355 -2.15 0.042 -2.232074 -.044608 _cons | 677.9564 239.1864 2.83 0.009 184.3 1171.613 -----------------------------------------------------------------------------Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 13
- 14. THE DANGER OF A “DATA MINING” APPROACH WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL 45° (predicted = actual) Actual vs. Predicted 60 50 R2 = .65 40 Predicted 30 20 10 0 0 10 20 30 40 50 60 Actual Source: Y and all Xs are independent random variables. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 14
- 15. THE DANGER OF A “DATA MINING” APPROACH THE MODEL IS USELESS: IT HAS NO PREDICTIVE VALUE Out-of-Sample Prediction 60 50 40 Best Prediction Predicted 30 20 10 0 0 10 20 30 40 50 60 Actual Source: Y and all Xs are independent random variables. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 15
- 16. HOW WILL WE KNOW IF WE BUILT A GOOD MODEL? HOW DO WE JUDGE OUR MODEL? • Predicts well • Predicts better than alternatives Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 16
- 17. POSITIVE DECISIONS BY AGENCY POSITIVE DECISION RATES (2007-2013) Positive Decision Rate HAS 95% NICE 72% CADTH 67% SMC 67% PBAC 62% Note: Based on 213 reviews of about 40 therapeutic areas between 2005 – May 2013 reviewed by NICE and at least one other major agency: SMC, PBAC, HAS, and CADTH. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 17
- 18. HOW WILL WE KNOW IF WE BUILT A GOOD MODEL? HOW DO WE JUDGE OUR MODEL? • Predicts well • Predicts better than alternatives • Provides actionable guidance for how to increase the likelihood of a positive reimbursement decision Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 18
Notas do Editor
- I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILDING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
- I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
- Agency agreement – main agencies
- Clinical assessment agreement, economic assessment agreement. NOTE for this presentation we did not have a joint table of agreement on the clinical AND economic in relationship to the decision.
- I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
- I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
- LESS SALEY
- LESS SALEY
- I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
- Number of reports: 78, 15, 39,39,42 (top to bottom)