How to understand and put into perspective a comparative effectiveness report

1. Demystifying Comparative Effectiveness Research: A Case Study Learning Guide Robert W. Dubois, MD, PhD Chief Medical Officer Cerner LifeSciences 1

2. Topics Opening thoughts Framework for reviewing a new CER report Case studies and lessons learned Randomized trial-incomplete answer Meta-analysis-misleading conclusion Observational study-incorrect change in clinical practice Final thoughts 2

3. Institute of Medicine Definition* “…The purpose of CER is to assist consumers, clinicians, purchasers and policy makers to make informed decisions that will improve health care at both the individual and population levels.” *IOM 2009 Initial National Priorities for Comparative Effectiveness Research 3

4. Not All Studies Are Created (or interpreted) Equally

5. 5

6. Proceed With Caution… 6

7. When the reports arrive…

8. It may feel like this…

9. Categorizing is good…

10. Understanding is even better…

11. Topics Opening thoughts How to approach CER 11

12. How to Approach CER

13. Step 1: Consider for Whom the Findings are Applicable An H1N1 study in school age kids may (or may not) help an internist with his/her adult population. A study of patients with severe rheumatoid arthritis may (or may not) apply to a population of milder patients. A study of diabetic control after one year may (or may not) apply to the longer term implications of a policy change. Will the study results generalize to your environment? 13

14. Questions for Step 1 Does the study resemble your population of interest (e.g., age, gender, SES, disease profile)? Does the clinical setting resemble your own (e.g., primary care, inner city)? Is the study conducted in the “real world” or in a highly controlled environment? Are the outcomes appropriate to your needs? 14

15. How to Approach CER

16. There are 3 Common Types of CER Randomized trials Meta-analysis Observational studies 16 Was the right design chosen?

17. 17

18. Does the design match the study question? RCT: may not meet your “real world” needs. Observational study: may have too many uncontrollable factors to be valid. Meta-analysis: just combining studies because they are “there” may not be appropriate. 18

19. Step 2: Consider Whether Aspects of the StudyDesign Might Affect the Results Does the study design match the question being asked? Was it carried out with adequate rigor? Each study design has its own issues to consider. One size does not fit all. 19

20. One size does not fit all…

21. One size does not fit all…

22. How to Approach CER

23. “How sure are we… really”

24. http://www.hasyudeen.com/2008/02/network-centric-competition-in-flat.html 24

26. Step 1-revisit the relevant population25

27. Can it Work? Randomized Controlled Trials (RCTs) Can an intervention work under certain controlled conditions? Characteristics: Planned experimental framework Defined treatment options Specified outcomes May be single or double blinded 26

28. RCT Design

29. RCTs Have Strengths and Weaknesses Strengths: Substantial internal validity Reduced likelihood of bias or confounding Gold standard for clinical research Pathway to FDA approval Weaknesses: May not generalize Modest size reduces ability to observe rare events. Placebo may not be the best “real world” comparator. Some patients leave the trial or “cross over” to the other therapy. 28

30. RCT Case Study: Cetuximab Clinical Situation Colorectal cancer is the 3rd most common cancer and cause of cancer death. Only an 11% survival if metastatic Treatment is not curative. Understanding of cancer genetics led to biomarker testing and targeted therapies. Cetuximab is an anti-epidermal growth factor receptor agent (EGFR). 29

31. Cetuximab RCT*: Was This the Full Story? 572 patients with mCRC Non-blinded RCT Cetuximab+ supportive care vs. supportive care alone Survival in Months *Jonker N Engl J Med 2007 30

32. Not the Full Cetuximab Story Basic science research suggested that the KRAS gene and its mutations could influence efficacy. Subgroup analysis performed based upon testing tissue samples Those with non-mutated gene had much greater benefit from cetuximab (5 months greater overall survival). 31

33. Not the Full Cetuximab Story FDA narrowed the drug’s indications to patients with a non-mutated gene. Recommendations to test patients for the genetic marker Therapy targeted to patients most likely to respond. Patients unlikely to respond avoid ill effects of the drug. 32

34. RCT Tips for the Consumer RCTs assess whether an intervention can work in a controlled environment. Although RCTs are viewed as the “gold standard,” the initial impression of the cetuximab study was incomplete. Multiple studies designed similarly would have likely yielded similar incomplete conclusions. A subgroup analysis based upon genetic markers yielded different results and conclusions. No results are permanent. 33

35. Topics Opening thoughts Framework for reviewing a new CER report (a.k.a. “Readers’ Guide” Case studies and lessons learned Randomized trial Meta-analysis-misleading conclusion Step 2: methods used (and not used) 34

36. Meta-Analysis “If one RCT is good, then more RCTs must be better…” Quantitative and statistical combination of study results Highest level of evidence Useful when different studies have different results 35

37. Bagshaw SM, Ghali WA. Acetylcysteine for prevention of contrast-induced nephropathy after intravascular angiography: a systematic review and meta-analysis. BMC Med. 2004;2(1):38. Metanalysis Clarifies the Benefit

38. Wilcken, MJA. 2007;186 (7):368-370. Metanalysis Clarifies the Benefits

39. Meta-Analysis Strengths Increases the effective sample size Provides statistically stronger conclusions Detects lower frequency events and more subtle distinctions Weaknesses Creates an impression of “truth” Easy to do wrong… If care is not taken, results may be invalid. 38

40. Meta-Analysis Case Study: Avandia Clinical Situation Blood sugar control reduces certain diabetic complications. But, oral drugs have been associated with an increased risk of heart disease (tolbutamide). TZDs seemed safer than sulfonylureas. Avandia approved by the FDA in 1999. 39

41. Original ArticleEffect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes Steven E. Nissen, M.D., and Kathy Wolski, M.P.H. N Engl J Med Volume 356(24):2457-2471 June 14, 2007 40

42. What was Done and Found? A meta-analysis examined the impact of Avandia on cardiac events. Out of 116 studies, 42 met the authors’ inclusion criteria. 15,565 Avandia patients 12,282 comparison patients 41

43. Avandia Adversely Impacts Heart Disease Compared with the placebo, estrogen plus progestin resulted in: Increased risk of heart attack Increased risk of stroke Increased risk of blood clots Increased risk of breast cancer Reduced risk of colorectal cancer Fewer fractures No protection against mild cognitive impairment and increased risk of dementia (study included only women 65 and older) Nissen SE, Wolski K. N Engl J Med. 2007;356:2457-2471.

45. Was This the Right Answer? “Thou shalt not combine heterogeneous studies…” Some studies had placebo and others active comparators. Some studies observed one arm longer than the other. Excluded “zero” event trials 44

46. Was This the Right Answer? The choice of statistical methods impacts the results. Re-analysis with an alternative approach showed no increased risk. Inclusion of the “zero event” studies also eliminated any statistical differences. New trial results (RECORD) showed no difference in deaths but an increase in heart failure. 45

47. Meta-analysis Tips for the Consumer Results are highly dependent on the studies included and excluded. Statistical methodology can impact study results. Nothing is permanent - emerging data may change the conclusions. 46

48. Topics Opening thoughts Framework for reviewing a new CER report (a.k.a. “Readers’ Guide” Case studies and lessons learned Randomized trial Meta-analysis Observational study-incorrect change in clinical practice Step 1: population differed in subtle ways from a more typical one Step 2: intervention and comparison groups were different 47

49. Observational Studies Answers the question: will it (likely) work (not can it work)? Examines the effects of treatment without formal randomization Performed prospectively or retrospectively 48

50. Observational Studies Framingham Study identified heart disease risk factors. Commonly uses existing “real world” databases Administrative or billing data Electronic health records 49

51. Observational Studies Strengths Assess health care in the “real world” Lower cost and faster to perform Large sample sizes feasible Hypothesis generating Weaknesses Administrative databases have minimal clinical detail and may contain errors. Uncontrolled design leads to potential bias or confounding. Subject to “data dredging” Cannot prove cause and effect 50

52. Observational Case Study: Hormone Treatment Clinical Situation Heart disease is the leading cause of death in women above age 50 Estrogen falls after menopause May account for the postmenopausal heat disease risk 51

53. What was Done and Found? Nurses’ Health Study (NHS) collected data from 48,470 women aged 30-55. Focused on the impact of hormones on outcomes Ongoing surveys assessed risk factors and health outcomes. Estrogen use associated with lower heart disease risk 52

54. Implications of the Nurses’ Health Study

55. Was This the Right Answer?

56. Was This the Right Answer? ?

57. The clinical trials were designed to test the effects of postmenopausal hormone therapy, diet modification, and calcium and vitamin D supplements on heart disease, fractures, and breast and colorectal cancer. 56

58. Women’s Health Initiative RCT with 27,347 post-menopausal women Mean age was 63.6 years. 57

59. Women’s Health Initiative Muddied the Waters… RCT with 27,347 post-menopausal women Mean age was 63.6 years. Study halted early. Increased risk of clots, breast cancer and stroke. Estrogen-progestin users: increased risk of heart disease Estrogen only users: no CV benefits. 58

60. Was This the Right Answer? ?

61. Why the Different Results? Theory 1: WHI population was older than the NHS cohort (started hormones much later) Theory 2: NHS women who took hormones were healthier Higher education Higher SES Leaner Lacked prior cardiac disease Theory 3: The NHS women who took hormones differed from those who did not 60

64. Topics Opening thoughts Framework for reviewing a new CER report (a.k.a. “Readers’ Guide” Case studies and lessons learned Final thoughts 63

65. Final Thoughts Different study types can offer different understandings. 64

66. Final Thoughts Different study types can offer different understandings. Results matter. Avandia sales fell after meta-analysis published. Hormone treatment plummeted after results of the WHI. Cetuximab use is growing. 65

67. Final Thoughts Different study types can offer different understandings. Results matter. Consider the potential impact upon policy making. Influence guidelines? Impact reimbursement? Stable clinical area or advancing rapidly? 66

68. Demystifying Comparative Effectiveness Research 67

69. Demystifying Comparative Effectiveness Research 68

70. Demystifying Comparative Effectiveness Research: A Case Study Learning Guide Reader’s Guide Check Lists Robert W. Dubois, MD, PhD Chief Medical Officer Cerner LifeSciences 69

71. How to Approach CER

72. Use Check Lists With Caution The lists are not all inclusive. A “yes” to all is not all that you need… All items are not created equally. 71

73. Step 1: Are the Findings Applicable?

74. Step 2: Design Affect the Results?

75. Step 2: Design Affect the Results?

76. Step 2: Design Affect the Results?

77. Step 3: Findings Change With New Research?