Webinar presented for the Mendeley webinar series: https://www.elsevier.com/solutions/mendeley/support/webinars?commid=253925 on reproducibility and research integrity.

1. Maintaining the Integrity of the Scholarly Record: It
Takes a Village!
William Gunn, Director of Scholarly Communication
April 13, 2017
@mrgunn

2. It takes a village
Librarians
Platforms
Funders
Publishers
Authors

3. ALS:TDI: Re-tested 70+ drugs from 221 independent
studies 1
➜ 0 reproduced
NINDS: Sponsored replication of 12 spinal cord injury
studies
➜ 2/12 fully reproduced 3
Bayer: Conducted in-house target validation studies
➜ 14/67 reproduced 4
Amgen: Attempted to reproduce 53 “landmark”
oncology publications
➜ 6/53 reproduced 5
Reproducibility Project: Cancer Biology:
Replications of 29 high-impact cancer biology studies
➜ 2/5 replicated, so far6

4. Publication
Bias
Problems identified
P-hacking
Bad stats
HARKing
Small “N”

5. Data
Sharing
Potential solutions identified
Negative
data
Unique IDs
Registered
Reports
Bigger “N”

6. Elsevier is already doing a number of these
Publishing replication studies
Creating a special article type for a replication study
Publishing “data papers” (1,2)
Hosting data
Linking clinical trials to original study registration
Introducing STAR Methods
Partnering to support reproducibility: Reproducibility Initiative, Reproducibility
Project: Cancer Biology, Resource Identification Initiative, Humboldt Advanced
Data & Text Center

7. Publishing replications & promoting reproducibility
Reproducibility Project: Cancer Biology
Registered Reports
Mandatory replication policy in Energy Economics and Review of Energy
Dynamics
Scientific Checklist in Biochemical Pharmacology
Resource Identification Initiative
CFP in Energy Economics
Linking clinical trials to the trial registrations
Launching journals devoted to Negative Results

8. Reproducibility Project: Cancer Biology
All registered reports published, 5/29 studies finished
Paper Conclusion Focus of key experiment Replication results
Sirota, M. et al. Sci. Transl. Med. 3, 96ra77 (2011)
Public gene expression data can identify unintuitive uses for
old drugs
Growth of tumours treated with an
anti-ulcer drug
Substantially reproduced
Sugahara, K. N. et al. Science 328, 1031–1035 (2010)
A tumour-penetrating peptide enhances the effects of
cancer drugs
Growth of peptide-treated tumours Not reproduced
Willingham, S. B. et al. Proc. Natl Acad. Sci. USA 109, 6662–6667
(2012)
Blocking contact between CD47 and another protein inhibits
tumour
Growth and metastasis of treated
tumours
Uninterpretable
Delmore, J. E. et al. Cell 146, 904–917 (2011) Blocking a protein sequence damps down pro-cancer genes
Gene expression in treated cells;
growth of treated tumours
Substantially reproduced
Berger, M. F. et al. Nature 485, 502–506 (2012)
Sequencing reveals gene that is frequently mutated in
melanoma and accelerates growth
Tumour formation in cells carrying
mutations
Uninterpretable
http://www.nature.com/news/cancer-reproducibility-project-releases-first-results-1.21304

9. Registered Reports in 7 Elsevier journals
Cortex Editorial
Drug and Alcohol Dependence Editorial
Human Movement Science Editorial
International Journal of Psychophysiology Editorial
Journal of Research in Personality Editorial
NFS Journal Announcement
The Leadership Quarterly Editorial

11. Linking clinical trials to publications

12. Hosting data & promoting reuse
Mendeley Data – Upload and share research data for free
Genomics Data & Data in Brief – Examples of successful data
publication journals
Elsevier Data Search – An index of research data across the web.

16. But back in the village…
New Negatives in Plant Science was discontinued due to few submissions.
CFP in Information Systems resulted in only two submissions
Less than half of clinical trials are published meaning there’s nothing to link to.
Funders don’t want to fund replication studies
Compliance with checklists isn’t monitored
Data sharing mandates aren’t enforced
And… there’s a persistent perception that publishers don’t want to publish
them.

17. More on researcher perception of replication
We interviewed 125 researchers to understand the changing appetite for publishing replication studies,
identify drivers and barriers to publication, and explore ways Elsevier can foster trustworthy science. They
said:
“Unless the original study appears to be flawed in some important way, why dedicate the time and
resources to an experimental program which appears to be well executed and well interpreted?”
“Replication studies in my opinion are becoming more and more important, especially since there is a big
pressure from the different journals to only publish new results…The thing that the journals do not want to
do is not to publish work that disproves [prior] work, but work that confirms [prior] work.”
“I think that publishers should explicitly say that replication is a good thing and that papers won't be
rejected on the basis of this”
“It might be a good idea to provide a special type of communication called "replication and verification" as
an avenue of publishing.”
Our messages are clearly not getting out there!

18. Just as with raising a healthy child, improving the
trustworthiness of research requires everyone to play
a part!

20. www.mendeley.com
william.gunn@elsevier.com
@mrgunn