O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Data collection methods to improve reproducibility

725 visualizações

Publicada em

"Reproducibility, data collection, and laboratory management technologies" - Louis Culot, CEO of Biodata

Slides from Shaking It Up: Challenges and Solutions in Scholarly Information Management, San Francisco, April 22, 2015

Publicada em: Ciências
  • Sharpen your mind with brain pill. learn more info.. ▲▲▲ https://tinyurl.com/brainpill101
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Seja a primeira pessoa a gostar disto

Data collection methods to improve reproducibility

  1. 1. Data collection methods to improve reproducibility Louis Culot, CEO, BioData
  2. 2. Published 1977 Main thesis: Running = Good ! Fix dies at age 52 (running) In 1984. Two camps: Running = Good ! Running = maybe good… moderation too much of a good thing?
  3. 3. Conclusions  The findings suggest a U-shaped association between all-cause mortality  and dose of jogging as calibrated by pace, quantity, and frequency of jogging. Light  and moderate joggers have lower mortality than sedentary nonjoggers, whereas  strenuous joggers have a mortality rate not statistically different from that of the  sedentary group. UHow much running Bad Bad Great ! Schnohr, Peter, et al. "Dose of jogging and long-term mortality: the  Copenhagen City Heart Study." Journal of the American College of Cardiology  65.5 (2015): 411-419.
  4. 4. Why Jogging May Be Better For Your Health Than Running So the connection was this: Joggers of mild and moderate intensity had a lower risk of  death than the strenuous joggers. In fact, the lowest mortality risk was that of the  mild intensity joggers. The fast-paced joggers had about the same rate of mortality as  sedentary people. This suggests that there may be an upper limit to in vigorous  exercise, after which the benefits fall off.
  5. 5. A recent study reported that joggers who exercise strenuously have the same life expectancy as people who do barely any exercise at all. But the author has now admitted he hasn't actually proved this. But, although the overall  number of people studied  was large, the number of  strenuous joggers was not.  Only 36 people fitted the  strenuous jogging category  - two of them had died.
  6. 6. State of Reproducibility -Amgen study: Only 6 of the 53 studies were reproduced (about 11%).1 -Bayer study: Only 14 out of the 67 projects (about 21%).2 -From 28% to 18% for clinical trials 2006-2007 – 2008-2010.3 ”…poor training, an emphasis on provocative conclusions in papers, a dearth of experimental details, and an overemphasis on publications in high-impact journals” – Francis Collins, NIH Director 1. Begley, C. Glenn, and Ellis, Lee M., "Drug development: Raise standards for preclinical cancer research." Nature 483.7391 (2012): 531-533 2. Prinz, Florian, Thomas Schlange, and Khusru Asadullah. "Believe it or not: how much can we rely on published data on potential drug targets?." Nature reviews Drug discovery 10.9 (2011): 712-712. 3. Nature Reviews Drug Discovery 10, 328-329 2011
  7. 7. State of Reproducibility -Amgen study: Only 6 of the 53 studies were reproduced (about 11%).1 -Bayer study: Only 14 out of the 67 projects (about 21%).2 -From 28% to 18% for clinical trials 2006-2007 – 2008-2010.3 ”…poor training, an emphasis on provocative conclusions in papers, a dearth of experimental details, and an overemphasis on publications in high-impact journals” – Francis Collins, NIH Director 1. Begley, C. Glenn, and Ellis, Lee M., "Drug development: Raise standards for preclinical cancer research." Nature 483.7391 (2012): 531-533 2. Prinz, Florian, Thomas Schlange, and Khusru Asadullah. "Believe it or not: how much can we rely on published data on potential drug targets?." Nature reviews Drug discovery 10.9 (2011): 712-712. 3. Nature Reviews Drug Discovery 10, 328-329 2011
  8. 8. Examples of Recommendations: “We recommend the following steps to change the culture of oncology research and improve the relevance of translational studies” (Begley and Ellis): There must be more opportunities to present negative data. It should be the expectation that negative preclinical data will be presented at conferences and in publications. Preclinical investigators should be required to report all findings, regardless of the outcome. To facilitate this, funding agencies, reviewers and journal editors must agree that negative data can be just as informative as positive data. Journal editors must play an active part in initiating a cultural change. There must be mechanisms to report negative data that are accessible through PubMed or other search engines. There should be links to journal articles in which investigators have reported alternative findings to those in an initial (sometimes considered landmark) publication. One suggestion is to include 'tags' that report whether the key findings of a seminal paper were confirmed. Other: Reporting unethical behavior, dialog between clinicians and patients and scientists; universities prioritizing teaching in tenure decisions (research is too “high stakes”). Better access to technologies – e.g., new cell lines, capabilities for genetic characterization of new tumor cell lines and xenografts.
  9. 9. “5 Practical Points to Consider” Poland, C. A., Miller, M. R., Duffin, R., & Cassee, F. (2014). The elephant in the room: reproducibility in toxicology. Particle and fibre toxicology, 11(1), 42. 1. Be specific. 2. Characterize, Characterize, Characterize 3. Use statistics to question your results rather than simply confirming a theory 4. Write your hypothesis in advance of running the experiment. 5. Take a ‘belts and suspenders’ approach to confirming findings.
  10. 10. “5 Practical Points to Consider” Poland, C. A., Miller, M. R., Duffin, R., & Cassee, F. (2014). The elephant in the room: reproducibility in toxicology. Particle and fibre toxicology, 11(1), 42. 1. Be specific. For example, what is your sample? If it’s graphene, then is it actually graphene i.e. a monolayer, or is it few layer graphene, or graphite platelets? If they are ambient particles, where and how were they collected (date, time, weather conditions etc.)? State this, it is important - later studies may not replicate your results simply due to inadvertently testing the non-similar materials i.e. comparing apples and oranges. =?
  11. 11. “5 Practical Points to Consider” Poland, C. A., Miller, M. R., Duffin, R., & Cassee, F. (2014). The elephant in the room: reproducibility in toxicology. Particle and fibre toxicology, 11(1), 42. 1. Be specific. 2. Characterize, Characterize, Characterize Only with independent characterisation do you truly know what you are testing and allow others to reproduce your findings. …. characterise in relation to your hypothesis or research question - do not characterise because there is a default list of parameters that you need to check. =? + 1. Sorge, Robert E., et al. "Olfactory exposure to males, including men, causes stress and related analgesia in rodents." Nature methods 11.6 (2014): 629-632. JAX CD34 JAX CD34
  12. 12. “5 Practical Points to Consider” Poland, C. A., Miller, M. R., Duffin, R., & Cassee, F. (2014). The elephant in the room: reproducibility in toxicology. Particle and fibre toxicology, 11(1), 42. Ask yourself, is barely statistically significant really biologically relevant? Can you support/validate an entire hypothesis or more controversially, a provocative statement using P<0.05?    1. Be specific. 2. Characterize, Characterize, Characterize 3. Use statistics to question your results rather than simply confirming a theory Revisit the power of negative results
  13. 13. The Economist ‘Trouble at the Lab’ 2013 Oct 19; available at http://go.nature. com/dstij3
  14. 14. The Economist ‘Trouble at the Lab’ 2013 Oct 19; available at http://go.nature. com/dstij3
  15. 15. The Economist ‘Trouble at the Lab’ 2013 Oct 19; available at http://go.nature. com/dstij3
  16. 16. The Economist ‘Trouble at the Lab’ 2013 Oct 19; available at http://go.nature. com/dstij3 “The root of all superstition is that men observe when a thing hits, but not when it misses.” …Francis Bacon
  17. 17. “5 Practical Points to Consider” Poland, C. A., Miller, M. R., Duffin, R., & Cassee, F. (2014). The elephant in the room: reproducibility in toxicology. Particle and fibre toxicology, 11(1), 42. 1. Be specific. 2. Characterize, Characterize, Characterize 3. Use statistics to question your results rather than simply confirming a theory 4. Write your hypothesis in advance of running the experiment. It should be clear from the experimental design how you have tried to disprove this. If your theory is robust it should survive your best efforts to challenge it.
  18. 18. “5 Practical Points to Consider” Poland, C. A., Miller, M. R., Duffin, R., & Cassee, F. (2014). The elephant in the room: reproducibility in toxicology. Particle and fibre toxicology, 11(1), 42. 1. Be specific. 2. Characterize, Characterize, Characterize 3. Use statistics to question your results rather than simply confirming a theory 4. Write your hypothesis in advance of running the experiment. 5. Take a ‘belts and suspenders’ approach to confirming findings. Replicate time points, dosages, protocols, etc. IC50 = 3.25 μl/mg vs vs Researcher 1 Researcher 2 Researcher 3
  19. 19. How labs use Labguru - Biospecimen management / sample tracking. - Project and experiment planing. - Sharing of data (internally and externally). - Managing collaborations. - Communicating with funders. - In-house repository for raw data and analysis.
  20. 20. What does it enable ? Science is struggling with reproducibility Scientific traceability 0 1 Ability to build your own and others’ work Communication as part of the experiment 0 2 0 3
  21. 21. It starts with how scientists manage their research. What affects reproducibility? It starts with scientists 1. Protocol Management: Lost protocols, inconsistent protocol updates, lab members using different protocols, and a lack of protocol – experimental and materials linkage. 2. Reagent and Specimen Details: Reagent – experiment coupling. Reagent conditions. Location of reagents. Inconsistent procedures for reagent synthesis. Genealogy. 3. Access to negative results: Often more reliable, and containing valuable information to develop new hypotheses. 4. Experimental Design: Disconnected data, lack of scientific context, and an incomplete picture make it hard for other scientists to reproduce work. “a problem well put is half solved.”… John Dewey
  22. 22. 1. Protocol Management • Lost protocols • Inconsistent protocol updates • Lab members using different protocols • Lack of protocol - experimental linkage The problem: What affects reproducibility? • Keep track of all protocols used • Designate & protocols between lab members • Keep a record of what versions were used for which experiments • Annotate protocols as they are executed. The solution: • Protocol Versioning • Built in protocol versioning and tracking system • Link to experiments for full scientific context The Labguru Approach:
  23. 23. 2. Reagent and Specimen Details • Reagent – experiment coupling • Reagent conditions • Specimen meta-data and provenance. • Location, origin, date, and handling. The problem: What affects reproducibility? • Record & track all successful reagents and specimens along with their metadata • Link specific specimens & reagents back to individual experiments • Identify problems from experimental records The solution: • Specimen and Reagent Tracking • Purchased materials are linked to experiments which use them • Smart modules for plasmids, cell lines bacteria, tube storage, and more The Labguru Approach:
  24. 24. 3. Access to Negative Results • Negative results often ‘lost’ over time in laboratories. Never published. The problem: What affects reproducibility? • Ready access to all project and experiments regardless of outcome. • Ability to back-test and build new hypotheses. • Increase ability to rescue failed experiments. The solution: • All data available within a lab (under PI control). • Data can be selectively shared and pushed to public repositories. • Simple ability to save “all data” in one place (including linking to big data, such as genomics and images). The Labguru Approach:
  25. 25. 4. Experimental Design • Disconnected data • Lack of scientific context • Incomplete picture • Loss of data when someone leaves the lab The problem: What affects reproducibility? • Begin with hypothesis testing. • Create detailed mini-reports that encompass all relevant data for an experiment • Connect related experiments and assets/materials. • Selectively reproduce “difficult” parts of experiments. The solution: • Platform for Searchability and Collaboration • Access data based on researcher, materials used, project, keywords • Integrated commenting and activity feeds increase accessibility The Labguru Approach:
  26. 26. Versioning Tracking Sharing history Provenance Genealogy Vendors Experiments & Protocols Calibration Experiments Vendor Results (e.g., drift) Change history Duplicates Results Contributors
  27. 27. The Digital Laboratory Ecosystem Labguru Lab Logistics/Notebook
  28. 28. The Digital Laboratory Ecosystem Labguru Lab Logistics/Notebook Physical Materials/Specimens Equipment Big data servers Genomics Imaging
  29. 29. The Digital Laboratory Ecosystem Labguru Lab Logistics/Notebook Physical Materials/Specimens Equipment Big data servers General Repositories Domain Repositories Genomics Imaging Manuscripts
  30. 30. Please contact us with any questions Visit labguru.com for a 30 day free trial of Labguru THANK YOU FOR COMING!

×