A presentation by Robert Kiley, head of digital services, Wellcome Trust and Neil Jacobs, head of open science and research lifecycle, Jisc.

1. Leadership conference 2018

2. Open science: what does
success look like, and how
would we know?
Robert Kiley, Head of Open Research, Wellcome
r.kiley@wellcome.ac.uk
Neil Jacobs, Head of Scholarly Communications Support, Jisc
n.Jacobs@jisc.ac.uk
Slides made available under http://creativecommons.org/licenses/by/4.0/

3. Open science* is…
1. Open access to publications
2. Wide sharing of data and other research outputs, and
3. Eschewing intellectual property rights (IPRs).etc
And can be practiced by
• Individual researchers
• Projects
• Institutions
• Collaborations
• Disciplines
• Research-as-a-whole
* Using term science – but refers to all research activities across all disciplines

4. Claims are made about the benefits
of Open Science
• Improves the robustness and verifiability of research by exposing it to wider
scrutiny
• Makes research more visible and accessible to other researchers, enabling
them to build on it and improve the quality of their research
• Increases the impact of research beyond the academic sector, contributing to
social and economic benefits
• Enables citizens to engage with research, and so increase trust in it, and its
legitimacy, as well as making it more relevant
• Supports collaboration, including international collaboration, at a time when
research needs to address global challenges

5. …but evidence to support these
clams is patchy

6. (though existing closed model is not
especially effective)
• Despite being a growing
burden to society, most
neurological diseases lack
effective treatments /
therapies
• In US, estimated cost of
neurological diseases is
$800 billion per year
• Diseases of the brain
predicted to surpass cancer
as the second leading cause
of death in Canada by 2040

7. …but lack of clear evidence that open
science works means that…
• Unclear how funders should best intervene for
maximum return on investment
• It leaves an apparent tension between open
science and commercialisation, especially on IP
• There is no clear way to assess whether open
science is, in fact, realising the benefits claimed
for it

8. And so a project to address these
issues
• This led Gates Foundation, Wellcome and BEIS (later Research England and
Jisc) to fund a project (led by Richard Gold, McGill) which seeks to:
1. Define what success looks like in open science
2. Develop a series of quantitative and qualitative indicators
3. Develop a codebook, allowing interested parties to start measuring the impact of
open science
4. Encourage take-up, and community of practice using the codebook
Ambition is to have an internationally recognised set of indicators that can be
adapted for local circumstances, but which give some consistency in assessing the
success and benefits of the global move to digital, open research

9. Phase 1- What success looks like
1) Increased quality and efficiency of scientific outputs;
2) Accelerated innovation and impact;
3) Increased trust and accountability of the research enterprise;
4) Increased equity in research;
5) Better opportunities and recognition of early career researchers
and youth;
6) Positive economic impact

10. Phase 2- Developing indicators
• 60 potential indicators identified
• Quantitative, based on existing or new data sources
• Qualitative, to be developed as standard protocols
• Examples include:
• FAIR Data Compliance Rate
• Diversity of users / readers (country)
• H-Index of open-access researchers (acknowledging concerns…)
• Funding for secondary use of data / experiment reproduction
• Trackability of the providence of research data
• Shannon Entropy (diversity in science)
• Research reproducibility (composite indicator)
• Increased innovation originating from ECRs who are practicing Open Science

11. 11
3. Knowledge and discoveries are shared, accessed and used in a
manner that maximises health benefit.
3b. Research outputs are findable, accessible, interoperable and re-usable (FAIR).
Description: The application of findability, accessibility, interoperability, and reusability (FAIR) principles help make research open,
maximising added-value gained by contemporary, formal scholarly digital publishing.
Outcome Indicator:
Proportion of Wellcome-
published research
outputs that adhere to
FAIR principles
Source: EPMC, Uber Dimensions

12. Eating our own dog food?
• All the benefits of open science…
• Quality
• Impact
• Engagement
• Trust, etc
• …are equally relevant to meta-science.
• Should the indicators of the success and benefits of open science themselves be
open and FAIR?
• “The data, metadata and methods that are relevant to research evaluation,
including but not limited to citations, downloads and other potential indicators of
academic re-use, should be publicly available for independent scrutiny and
analysis by researchers, institutions, funders and other stakeholders.” Integrated
advice of the EC Open Science Policy Platform Recommendations, May 2018

13. Questions
13

14. Robert Kiley
Head of Open Research, Wellcome
r.kiley@wellcome.ac.uk
Thankyou
Neil Jacobs
Head of Scholarly Communications Support, Jisc
n.Jacobs@jisc.ac.uk