Presented at the Leiden Bioscience Lecture, 24 November 2016, Reproducibility, Research Objects and Reality
Over the past 5 years we have seen a change in expectations for the management of all the outcomes of research – that is the “assets” of data, models, codes, SOPs, workflows. The “FAIR” (Findable, Accessible, Interoperable, Reusable) Guiding Principles for scientific data management and stewardship have proved to be an effective rallying-cry. Funding agencies expect data (and increasingly software) management retention and access plans. Journals are raising their expectations of the availability of data and codes for pre- and post- publication. It all sounds very laudable and straightforward. BUT…..
Reproducibility is a R* minefield, depending on whether you are testing for robustness (rerun), defence (repeat), certification (replicate), comparison (reproduce) or transferring between researchers (reuse). Different forms of "R" make different demands on the completeness, depth and portability of research. Sharing is another minefield raising concerns of credit and protection from sharp practices.
In practice the exchange, reuse and reproduction of scientific experiments is dependent on bundling and exchanging the experimental methods, computational codes, data, algorithms, workflows and so on along with the narrative. These "Research Objects" are not fixed, just as research is not “finished”: the codes fork, data is updated, algorithms are revised, workflows break, service updates are released. ResearchObject.org is an effort to systematically support more portable and reproducible research exchange
In this talk I will explore these issues in data-driven computational life sciences through the examples and stories from initiatives I am involved, and Leiden is involved in too including:
· FAIRDOM which has built a Commons for Systems and Synthetic Biology projects, with an emphasis on standards smuggled in by stealth and efforts to affecting sharing practices using behavioural interventions
· ELIXIR, the EU Research Data Infrastructure, and its efforts to exchange workflows
· Bioschemas.org, an ELIXIR-NIH-Google effort to support the finding of assets.
Reproducibility, Research Objects and Reality, Leiden 2016
1. Reproducibility,
Research Objects
and Reality
Professor Carole Goble
The University of Manchester, UK
Software Sustainability Institute, UK
ELIXIR UK,
FAIRDOMAssociation e.V.
carole.goble@manchester.ac.uk
University of Leiden,The Netherlands, 24 November 2016
2. Acknowledgements
• Dagstuhl Seminar 16041 , January 2016
– http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041
• ATI Symposium Reproducibility, Sustainability and Preservation , April 2016
– https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/
– https://osf.io/bcef5/files/
• CTitus Brown
• Juliana Freire
• David De Roure
• Stian Soiland-Reyes
• Barend Mons
• Tim Clark
• Daniel Garijo
• Norman Morrison
• Katy Wolstencroft
Phil Bourne
Natalie Stanford
Jacky Snoep
Stuart Owen
Marco Roos
Kristina Hettne
AlanWilliams
Sean Bechhofer
Ian Fore
Rafael Jimenez
…. And many more
Michael Crusoe
Paul Groth
Niall Beard
4. Motivation: Knowledge Turning
research infrastructures
• Computational tools
• Sharing platforms
• Knowledge
Exchange
• Reproducible
research
• Software and data
practices
• Policies
[Josh Sommer, for the picture]
14. Research Infrastructure for
FAIR Management and Sharing of
Data, Operating Procedures, Model
For Systems and Synthetic Biology
Projects
Research Infrastructure for
FAIR Data for Life Sciences in
Europe
Data-Driven Science
15.
16. design
cherry picking data, random seed
reporting, non-independent bias, poor
positive and negative controls, dodgy
normalisation, arbitrary cut-offs,
premature data triage, un-validated
materials, improper statistical analysis,
poor statistical power, stop when “get to
the right answer”, software
misconfigurations misapplied black box
software
reporting
incomplete reporting of software configurations, parameters & resource
versions, missed steps, missing data, vague methods, missing software
Empirical Statistical Computational
V. Stodden, IMS Bulletin (2013)
Reproducibility and reliability of biomedical
research: improving research practice
17. “When I use a word," Humpty Dumpty
said in rather a scornful tone, "it means
just what I choose it to mean - neither
more nor less.”
Carroll, Through the Looking Glass
re-compute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle
redo
robustness
tolerance
verificationcompliancevalidation assurance
remix
18. Scientific publications goals:
(i) announce a result
(ii) convince readers its correct.
Papers in experimental science
should describe the results and
provide a clear enough protocol to
allow successful repetition and
extension.
Papers in computational science
should describe the results and
provide the complete software
development environment, data
and set of instructions which
generated the figures.
VirtualWitnessing*
*Leviathan and theAir-Pump: Hobbes, Boyle, and the
Experimental Life (1985) Shapin and Schaffer.
Jill Mesirov
David Donoho
21. Repeatability:
“Sameness”
Same result
1 Lab
1 experiment
Reproducibility:
“Similarity”
Similar result
> 1 Lab
> 1 experiment
why the differences?
https://2016-oslo-
repeatability.readthedocs.org/en/latest/repeatability-discussion.htm
Validate
Verify
22. Method Reproducibility
the provision of enough detail about
study procedures and data so the
same procedures could, in theory or in
actuality, be exactly repeated.
Result Reproducibility
(aka replicability)
obtaining the same results from the
conduct of an independent study
whose procedures are as closely
matched to the original experiment
as possible
Goodman, et al ScienceTranslational Medicine 8 (341) 2016
Validate
Verify
24. reviewers want additional work
statistician wants more runs
analysis needs to be repeated
post-doc leaves,
student arrives
new/revised datasets
updated/new versions of
algorithms/codes
sample was contaminated
better kit - longer simulations
new partners, new projects
Personal & Lab
Productivity
Public Good
Reproducibility
25. Computational “Datascopes”
Methods
techniques, algorithms,
spec. of the steps, models
Materials
datasets, parameters,
algorithm seeds
Instruments
codes, services, scripts,
underlying libraries,
workflows, ref datasets
Laboratory
sw and hw infrastructure,
systems software,
integrative platforms
computational environment
28. Active Instrument
Byte level preservation
Reproduce by RunningReproduce by Reading
Archived Record
Prepare to repair
ELNs
Markup Languages
Reporting Guidelines
Common Formats
Community
vocabularies
33. Workflow Preservation and Exchange
Experiments
Workflows &Workflow Runs
Workflow Commons
Third Party Services
Scattered resources
34. Workflow Preservation and Exchange
Experiments
Workflows &Workflow Runs
Workflow Commons
Third Party Services
Scattered resources
Rich descriptions
Prepare to Repair
35. Standards-based metadata framework for bundling resources
with context
Citable Reproducible Packaging
Metadata for bundling resources scattered and stored somewhere else
36. Container
Research Object in a nutshell
Packaging content & links:
Zip files, BagIt, Docker images
Catalogues & Commons Platforms:
FAIRDOM, myExperiment
37. Manifest
Construction
Aggregates
link things together
Annotations
about things & their
relationships
Container
Research Object in a nutshell
Manifest
Description
Dependencies
what else is
needed
Versioning
its evolution
Checklists
what should
be there
Provenance
where it
came from
Identification
locate things
regardless where
id
Packaging content & links:
Zip files, BagIt, Docker images
Catalogues & Commons Platforms:
FAIRDOM, myExperiment
38. Manifest
Construction
Aggregates
link things together
Annotations
about things & their
relationships
Container
Research Object Profile forWorkflows…
Manifest
Description
Identification
locate things
regardless where
Minimum information
for one content type
Common properties
among content
types
39. Research Object Profile forWorkflows…
Manifest
Description
Minimum information
for one content type
Common properties
among content
types
40. Belhajjame et al (2015) Using a suite of ontologies for preserving workflow-centric research objects,
JWeb Semantics doi:10.1016/j.websem.2015.01.003
Hettne KM, et al (2014), Structuring research methods and data with the research object model: genomics workflows as a
case study. J. Biomedical Semantics 5: 41
Workflow Research Object Bundles
exchange, portability and maintenance
BagIt
workflows packaged into
various containers for sharing
Checksum
41. Workflow and Workflow Management System Zoo
https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems
42. bio.tools
A community led standard way
of expressing and running
workflows and command line
tools using containers
Ontologies for describing tools
and their inputs and outputs
Metadata framework for the
manifest versioning, file
integrity, more metadata
about the workflow
Workflow fragment containers
46. Systems Approach…
Multiple, interrelated assets, Multiple, dispersed repositories
Literature
SOPS
STANDARDS
versioning,
tracking:
provenance,
parameters,
citation
Operations
47. FAIR Data and Metadata Standards that
help to improve understanding and exchange….
Nicolas Le Novère, Babraham Institute, UK.
…researchers do not always use them....
48. … model reuse and reproducibility tricky…
Stanford et alThe evolution of standards and data management practices in systems
biology, Molecular Systems Biology (2015) 11: 851 DOI 10.15252/msb.20156053
50. What methods are been used to determine
enzyme activity?
What SOP was used for this sample?
Where is the validation data for this model?
Is there any group generating kinetic data?
Is this data available?
Track versions of my model
Whats the relationship between the data and
model?
Which data belong to
which publications?
FAIR
54. ….organised in Investigation, Study, Assay/Analysis format
….registered using Just Enough Results description.
Just Enough
Results Model
Common elements
55. ….organised in Investigation, Study, Assay/Analysis format
….registered using Just Enough Results description.
Uploaded into the
FAIRDOM Store
Linked to entry
in Public Archive
Linked to entry in
Project store
56. ... aggregating catalogue
metadata across repositories, retain context-> reproduce, reuse
Local Stores
External
Databases
Publishing services
Secure
Stores
Model
Resources
57. … in situ reproducible models
metadata annotation against standards
model validation, comparison and simulation
SBML Model simulation
Model comparison
Model versioning
Reproducing simulations
[Jacky Snoep, Dagmar Waltemate, Martin Peters, Martin Scharm]
59. Research Objects
• Link
• Nest
• Span
• Bundle
• Snapshot
Systematic, Standards-
based metadata
framework for logically
and physically bundling
resources with context
• Exchange
• Reproduce
• Release packages
60. Reproducible Exchange and Publishing
and better credit
Author List: Joe Bloggs; Jane Doe
Title: My Investigation
Date: September 2016
DOI: https://doi.org/10.15490/seek##
information travels with the data and models
61. How do we do? Pretty well.
Reproducibility window. But that’s ok!
• Can’t contain everything
– Pesky Internet in a Box
• Can’t automate everything
– Pesky people
• Can’t fix everything
– Pesky science
63. Samiul Hasan, GSK
Biocuration need in Pharma: Drivers from aTranslational Bioinformatics Perspective,
Poster S16
1st EASYMConference, Berlin 2016
Reality
64. Preparation pain. Goldilocks paradox.
[Norman Morrison]
replication hostility no funding, time, recognition, place to publish
resource intensive access to the complete environment
66. Using FAIRDOM my own
lab colleagues saw what I
was doing and called to
collaborate!
Jurgen Hannstra
Vrije Universiteit Amsterdam, Netherlands
Trust …
67.
68.
69. Half of researchers make research data available
so they can be used by another.
Most not experienced any direct benefits
nor experienced many bad effects.
Caveat:
shared but usable?
fake sharing
funder requirements
fear data will be
misused or
misinterpreted
journal requirements
good research practice
facilitate collaborations
enable validation and
replication
higher citation rates
time and effort
new collaborations
extra funding for cost of data prep
enhance their academic reputation
feedback on how other researchers were using
their data
taken into account in funding
taken into account in career
jeopardise future publications
its not ready to share
scrutiny scruples
answering questions
I won’t get credited
70. Metadata in by side effect
Tooling for annotations and checklist templates for different types of assay data.
Embed ontologies into
Excel templates
Excel spreadsheets enriched
with ontology annotations
Upload, extract metadata and register
http://www.rightfield.org.uk
Spreadsheet Ramps!!
71. Sharing by side effect …. libertarian paternalism
[Kristian Garza]
72. Finding and Citing by side effect
• Schema.org
• Structured
markup in web
pages
• Supported by
Content
Management
Systems
• Harvested by
search engines
• Builds snippets
and sidebars
Bioschemas.org
74. Big co-operative data-driven
science makes reproducibility
desirable but also means
dependency and change are to be
expected
Words matter.
50 Shades of Reproducibility.
form vs function
Reproducibility is not a end.
Beware zealots.
Amplify Side effects
Think Research Objects!