5. We have a problem…
Many scientific disciplines currently face a ‘reproducibility crisis’, whereby a
worryingly large proportion of results cannot be independently replicated.
(Stodden 2011, Ioannidis 2014, Parker and Nakagawa 2014, Parker et al. 2016)
6. In psychology and biomedical science, at least two-thirds of
published findings are estimated to be incorrect.
53 'landmark' studies
in cancer research
Only 6 studies (11%) were
successfully replicated
7. 100 studies in three psychology
journals using high-powered designs
and original materials when available
39% of effects rated to have
replicated the original result
In psychology and biomedical science, at least two-thirds of
published findings are estimated to be incorrect.
8. The reproducibility crisis in science
The case against science is straightforward: much of the scientific
literature, perhaps half, may simply be untrue. Afflicted by studies with
small sample sizes, tiny effects, invalid exploratory analyses, and flagrant
conflicts of interest, together with an obsession for pursuing fashionable
trends of dubious importance, science has taken a turn towards darkness.
(Horton 2015, The Lancet)
9. We don’t really know…
How serious is the problem in Ecology & Evolution?
10. There are only a handful of
replication efforts in E&E
Clint Kelly, UQÀM
11. BUT: they have devastating effects on the public perception of scientists
and promote mistrust and denial on topics such as evolution, climate
change, overfishing, pollution, etc.
Irreproducible results in E&E might not have the same consequences
for human welfare as in medicine and psychology…
17. The reproducibility crisis is also a credibility crisis
At a time of post-truth politics, widespread pollution, climate change,
and an increasing anti-vaccination movement,
science cannot afford a credibility crisis.
(Stodden 2011, Amstat News )
20. “Selection for high output leads to poorer methods and increasingly
high false discovery rates.”
(Smaldino and McElreath 2016, Roy Soc Open Sci)
21. • publication bias
• confirmation bias
• HARKing
• under-reporting
• selective reporting
• P-hacking
Poor practices that contribute to the replication crisis:
(Parker et al 2016, TREE)
22. • publication bias
• confirmation bias
• HARKing
• under-reporting
• selective reporting
• P-hacking
Poor practices that contribute to the replication crisis:
(Parker et al 2016, TREE)
23. Confirmation bias
= increase in false discoveries
The widespread human tendency to preferentially search for and interpret
observations as consistent with one's beliefs about how the world works.
25. P-hacking
A variety of practices that increase the odds of finding a statistically
significant result by, for instance, conducting multiple versions of an analysis
with different covariates, interactions, or subsets of the data.
30. Improving our research workflow
The sequence of activities & procedures at
different stages of a research project.
1. Question, hypotheses, predictions
2. Experimental and statistical design
3. Data collection
4. Data management (organization, cleaning)
5. Statistical analysis (using script)
6. Interpretation of results
7. Publication
preregistration}
31. Preregistration = making (and registering) key decisions about the
study design and data analysis early in the research lifecycle, before
collecting the data.
32. Registered Reports are a publication format where peer review occurs
prior to data collection and analysis.
Accepted papers then are virtually guaranteed publication in the
journal if the authors follow through with the registered methodology.
80 journals publish RRs
33. Does preregistration prevent additional data
collection / analyses?
No : non-preregistered measurements and analyses are allowed
but must be labelled as exploratory (rather than confirmatory) in
the publication.
34. Curbing the reproducibility crisis by improving our
research workflow
= reduce selective reporting and P-hacking
35. Improving our research workflow
The sequence of activities & procedures at
different stages of a research project.
1. Question, hypotheses, predictions
2. Experimental and statistical design
3. Data collection
4. Data management (organization, cleaning)
5. Statistical analysis (using script)
6. Interpretation of results
7. Publication
preregistration}
blinding, proper documentation}
37. Conduct double-blind studies in E&E
control
treatment
Blind protocols are uncommon in
the life sciences and nonblind
studies report higher effect sizes
and more significant p-values.
(Holman et al 2015 PLOS Biol)
38. Does the experimental design exclude the possibility of observer
and experimenter bias (e.g., by double blind protocol)?
Video recording facilitates blinding and is possible in most
well-devised experiments.
(Clark et al 2016 Trends Ecol Evol)
40. Curbing the reproducibility crisis by improving our
research workflow
= reduce selective reporting and P-hacking
= reduce confirmation bias, allow replication
41. Improving our research workflow
The sequence of activities & procedures at
different stages of a research project.
1. Question, hypotheses, predictions
2. Experimental and statistical design
3. Data collection
4. Data management (organization, cleaning)
5. Statistical analysis (using script)
6. Interpretation of results
7. Publication
preregistration}
open data & script}
blinding, proper documentation}
42. Editorial open data policies
Representative journals that require data archiving
•The American Naturalist
•Biological Journal of the Linnean Society
•Biology Letters
•BMC Ecology
•BMC Evolutionary Biology
•BMJ
•BMJ Open
•Ecological Applications
•Ecological Monographs
•Ecology
•Ecosphere
•Evolution
•Evolutionary Applications
•Frontiers in Ecology and the Environment
•Functional Ecology
•Genetics
•Heredity
…
43. = reduce selective reporting and P-hacking
= reduce confirmation bias
= lots of benefits (for individuals and society)
Why do it?
44. The sequence of activities & procedures at
different stages of a research project.
1. Question, hypotheses, predictions
2. Experimental and statistical design
3. Data collection
4. Data management (organization, cleaning)
5. Statistical analysis (using script)
6. Interpretation of results
7. Publication
preregistration}
open data & script}
blinding, proper documentation}
Curbing the reproducibility crisis requires
improving our research workflow
45. = reduce selective reporting and P-hacking
increase credibility
= reduce confirmation bias, allow replication
increase reproducibility
= reduce genuine errors and miscondut
increase reusability, accelerate discovery
Open Science
46. Thanks to Sandra Binning, Redouan Bshary, Hanna Kokko, Loeske Kruuk, Michael
Jennions and Rob Lanfear and the eco-ethology lab at UniNE for insightful discussions.
Image / illustration credits: A. Saego, Google Images@dom_roche