Amanda burls ppt teaching materials for publication bias
•Transferir como PPT, PDF•
5 gostaram•1,357 visualizações
Teaching materials explaining publication bias. For use with students with no prior knowlede or previous training in critical appraisal skills. It explains both failure to report and selective reporting. The notes beneath each slide give suggestions of how the materials can be used in class. The examples are real example e.g. a trial of activia yoghurt versus yoghurt without live probiotics for irritable bowel syndrome (IBS), Basic slides will take only 5 minutes in a presentation or the materials can be used in a full one hour interactive workshop. These were put together in 2016 for www.testingtreatments.org which helps explain a fair test of treatment. They are particularly useful for teaching the critical appraisal of systematic reviews.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Amanda burls ppt teaching materials for publication bias
Semelhante a Amanda burls ppt teaching materials for publication bias (20)
Último
Último (20)
Amanda burls ppt teaching materials for publication bias
- 1. Publication bias – teaching materials • The attached slides can be used to teach people about publication bias • There are notes beneath the slides with suggestions of how they might be used • Prepared by Professor Amanda Burls and available from http://openaccess.city.ac.uk/id/eprint/13488Amanda.Burls.1@city.ac.uk • Publication bias by Dr Amanda Burls is licensed under a Creative Commons Attribution 4.0 International License.
- 2. Who’s heard of friendly bacteria?
- 3. You are seeing a patient and she asks… “Should I buy Activia™ to help with my tummy pain and improve my constipation?” You have 1 minute to discuss with your neighbour what you would tell her? 1:000:590:580:570:560:550:540:530:520:510:500:490:480:470:460:450:440:430:420:410:400:390:380:370:360:350:340:330:320:310:300:290:280:270:260:250:240:230:220:210:200:190:180:170:160:150:140:130:120:110:100:090:080:070:060:050:040:030:020:01End
- 4. Which of the following is closest to your answer? 1. 2. 3. 4. 5. 6. 7. 3% 21% 0% 39% 18% 5% 13% 1. Try it - it can help some people 2. You need to ask a doctor a question like that 3. Yes, but any live yoghurt will do 4. No, there’s no evidence it helps 5. I don’t know 6. I don’t know, but it can’t do any harm 7. I don’t know, but I’ll look up the evidence and tell you next week
- 5. What is the evidence? • What sort of study design would be best for answering this question?
- 6. Randomised Control Trial (RCT) of live yoghurt for Inflammatory Bowel Syndrome (IBS) • P: Adults with IBS • I: Activia (2 pots per day) • C: Identical yoghurt with no live bacteria (2 pots per day) • O: Adequate symptom relief • T: 12 weeks • Funded by Danone • Undertaken by independent researchers at the University of Birmingham
- 7. RCT comparing live yoghurt with ordinary yoghurt IBS • The trial ended in 2005 • What do you think the results showed?
- 8. Results for primary outcome 2:001:591:581:571:561:551:541:531:521:511:501:491:481:471:461:451:441:431:421:411:401:391:381:371:361:351:341:331:321:311:301:291:281:271:261:251:241:231:221:211:201:191:181:171:161:151:141:131:121:111:101:091:081:071:061:051:041:031:021:011:000:590:580:570:560:550:540:530:520:510:500:490:480:470:460:450:440:430:420:410:400:390:380:370:360:350:340:330:320:310:300:290:280:270:260:250:240:230:220:210:200:190:180:170:160:150:140:130:120:110:100:090:080:070:060:050:040:030:020:01End
- 9. Results for primary outcome
- 10. One participant in trial finally wrote to the investigator when these results had not been published five years later. This was the reply…
- 11. Sent: 19 January 2010 15:45 To: Dr Amanda Burls Subject: RE: Yoghurt trial Dear Amanda, The trial is not yet in press - this is in part due to the much longer than anticipated further analysis of the data at the funders request. In summary this was a negative trial - although both groups demonstrated benefit, those in the active product group did not show greater benefit and at times the difference actually favoured the control product….
- 12. Publication bias – the tendency not to publish negative findings
- 13. The trial was finally published eight years later (after pressure to do so)...
- 14. However, publication bias is • Not only about not publishing • Can also be about reporting outcomes that were positive but not mentioning those that were not (even when they were the primary outcome) • Putting more positive results in the Abstract of the paper • Distorting the Conclusions
- 15. Let’s compare your conclusions to those in the paper
- 16. Let’s compare our implications for practice with those of the authors
- 17. A biased conclusion - great for Danone! • Activia was worse than the control yoghurt • But, let’s recommend fermented dairy products anyway! • To back up such a recommendation randomized control trials are needed showing that fermented dairy products help patients with IBS compared to not having them. • Otherwise we can’t exclude bias or regression to the mean (that people get better anyway, without treatment!)
- 18. Further slides for later classes on publication bias
- 19. Looking for bias in systematic reviews
- 20. A funnel plot Size of study Estimate of truth T r u t h
- 21. Funnel plots… • …are scatter plots of treatment effect estimated from individual studies (x axis) against a measure of each study’s sample size (y axis). • The precision of the estimates of the treatment effect increases as sample size increases. • Effect estimates from small studies scatter more widely at the bottom of the graph, with the spread narrowing among larger studies. • In the absence of bias the plot should resemble a symmetrical inverted funnel. Size of study Treatment effect
- 22. A funnel plot Size of study
- 23. A funnel plot Size of study
- 24. A funnel plot Size of study
- 25. Publication bias distorts results
- 26. A funnel plot Size of study Favours treatment Favours control
- 27. A funnel plot Size of study Favours treatment Favours control
- 28. Sources of asymmetry • Publication bias • Poor methodological quality of smaller studies • True heterogeneity i.e. size of effect differs according to study size • for example, due to differences in the intensity of interventions or differences in underlying risk between studies of different sizes • Chance
Notas do Editor
- Please feel free to use these materials in any way you want, re-mixing or just as they are. I have included examples of where I use them together with TurningPoint clickers for large groups with sample results of votes, but the same can be achieved by a show of hands if a safe learning environment is created.
- Engage the students by getting them to chat about the TV adverts they have seen for friendly bacteria and/or get them to say if they take Actimel or other probiotic “health” drinks.
- This can be friend or colleague rather than patient if your audience is not a clinical audience. The timer counts down 1 minute/
- This particular group voted after about four other examples showing them why they need to look for research evidence before giving advice to patients. When used at the beginning of a presentation most people vote for 6.
- The best primary study to address this question would be a randomized control trial and the best study design for looking for an answer would be an up-to-date high quality systematic review of randomized controlled trials. Students can be engaged by asking them if they think that there would be trials of such a question (most vote “no”).
- PICO stands for Population, Intervention, Comparator and Outcome The photo is of the intervention used in an RCT of Activia (not named in trial) funded by Danone This picture is also useful for discussion blinding and placebo effects. I don’t usually included the text that is grayed out above, I just say this at some point when going through the story. (I was a participant in the trial!)
- PICO stands for population, intervention, comparator and outcome The photo is of the intervention used in an RCT of Activia (not named in trial) funded by Danone This picture is also useful for discussion blinding and placebo effects. I don’t usually included the text that is grayed out above, I just say this at some point when going through the story. (I was a participant in the trial!)
- What do these results mean. Get participants to discuss in pairs. Get them to think both about the findings and to write a brief lay summary including implications for practice. Two minute timer. You can collect the summaries to talk about later. An even better way of doing this is to hand out cards with the result but half of them have the results reversed for active and control product. As shown on next slide. This enables you to explore the effect of prior beliefs and direction of effect with class in relation to what they wrote.
- Faked slide with control and active product results reversed. An even better way of doing this is to hand out cards with the result but half of them have the results reversed for active and control product. As shown on next slide. This enables you to explore the effect of prior beliefs and direction of effect with class in relation to what they wrote.
- I searched for the results of this trial every six months from 2006 onwards but could not find the results. Finally, I wrote to the principal investigator and this is the email she sent back!
- These slides I would tend to use on a subsequent teaching session – I wouldn’t introduce funnel plots to new learners until they had some confidence in critical appraisal, but I have included them in case you find them useful.
- I use these slides with a large bag of green and yellow counters where I get the students to estimate the proportion of counters that are green drawing different size samples. You can draw on a flipchart the estimate each time based on number of counters drawn out (and replaced) You end up creating a funnel plot around the true proportion. So that students can understand that the variation at the bottom is just due to chance and that the bigger the sample the less variation there is due to chance and that because it is due to chance it is equally as likely to underestimate as overestimate the true result and therefore the plot will be symmetrical.
- You can toggle backwards and forwards between this slide and the previous slide using up and down arrows to talk about how treatment effectiveness is distorted
- substantial benefit may be seen only in patients at high risk for the outcome that is affected by the intervention, and these high risk patients are usually more likely to be included in early, small studies (Davey Smith 1994, Glasziou 1995). In addition, small trials are generally conducted before larger trials are established, and in the intervening years standard treatment may have improved. Furthermore, some interventions may have been implemented less thoroughly in larger trials and, therefore, may have resulted in smaller estimates of the treatment effect (Stuck 1998). It has also been argued that funnel plot asymmetry may be an artefact (Irwig 1998), but simulation studies have shown that this will occur infrequently if the overall treatment effect is very large and the outcome of interest is rare (Sterne, unpublished). Finally, it is clearly possible that an asymmetrical funnel plot arises merely by the play of chance.