Jennifer Freeman Smith talks about increasing research transparency using the Open Science Framework | OSFair2017 Workshop Workshop title: Increasing Research Transparency using the Open Science Framework Workshop overview: Part of the challenge with making research more open and transparent is purely logistical. Where and how can the research be stored, organized, and shared most effectively when there are so many different tools, processes and policies in place? The OSF provides an open source, structured environment where researchers from all over the world, using their own tools and processes, can collaborate openly, transparently, and effectively. DAY 3 PARALLEL SESSION 8

1. cos.io | osf.io
Jennifer Freeman Smith, PhD
Transparency and Openness
Training Coordinator
Increasing Research
Transparency Using the Open
Science Framework

2. Improving Openness, Integrity, and
Reproducibility of Scientific Research
MISSION

3. Infrastructure Metascience
Community

4. Everything we do is free
and open.
COS Strategic Plan: https://osf.io/x2w9h

5. More at: http://cos.io/about/our-sponsors

6. How does transparency
improve science?

8. SOURCES OF ISSUES IN
REPRODUCIBILITY
• Methodological, statistical, and reporting
practices
• Structural and organizational practices
• Rarely, intentional scientific misconduct

9. WHAT IS REPRODUCIBILITY?
Computation Reproducibility:
If we took your data and code/analysis scripts and reran it,
we can reproduce the numbers/graphs in your paper
Methods Reproducibility:
We have enough information to rerun the experiment or
survey the way it was originally conducted
Results Reproducibility/Replicability:
We use your exact methods and analyses, but collect new
data, and we get the same statistical conclusion

10. If we seek to facilitate reproducibility,
replicability, extension, and reuse…

11. We need to move beyond description of
outcomes to description of process
or, better, sharing
actual process.

12. Open Access
Open Data
Open Materials
Open Data Cleaning Scripts
Open …
Open Workflow

13. OPEN WORKFLOW
• Increases process transparency
• Increases accountability
• Facilitates reproducibility
• Facilitates metascience
• Fosters collaboration
• Fosters inclusivity
• Fosters innovation
• Protects against lock-in: Open + Accessible

14. It takes some effort to organize your
research to be reproducible…the
principal beneficiary is generally the
author herself.
Jon Claerbout
Making
Scientific Contributions Reproducible
http://sepwww.stanford.edu/oldsep/matt/join/redoc/web/iris.html
“
”

15. Write
Report
Publish
Report
Search/
Discovery
Develop
Idea
Design
Study
Collect
Data
Store Data
Analyze
Data OS
F

16. OPEN SCIENCE FRAMEWORK
http://osf.io
Free, open source scientific commons

17. Collaboration, Documentation,
Archiving, Sharing

18. Put data, materials, and code on the OSF

19. Automatic file versioning

20. Now
Publish
Report
Search /
Discovery
Develop
Idea
Design
Study
Collect
Data
Store
Data
Analyze
Data
Write
Report

21. Future
Publish
Report
Search /
Discovery
Develop
Idea
Design
Study
Collect
Data
Store
Data
Analyze
Data
Write
Report

22. Why open your workflow?

23. WHY OPEN YOUR WORKFLOW?
• Improve reproducibility and replicability
• Increased efficiency
• Increases reuse and extension of knowledge
• Public data can be combined with private data
• Can influence scientists, entrepreneurs, policymakers,
citizens

24. OPEN DATA CHALLENGES
• How do we make our data accessible, understandable,
reusable?
• Which repository should I choose?
• Who owns the data? Do I have a copyright on the raw data
I collected?
• If I reuse data from someone else, do I have to offer them
co-authorship?
• How should privacy issues be addressed?

25. SHARING IS A CONTINUUM
• Data underlying just results reported in a paper
• Data underlying publication + information about other
variables collected
• Data underlying publication + embargo on full dataset
• All data collected for that study

26. Persistent
citable
identifiers

27. GUIDs make sharing simple
Arnold BF, van der Laan MJ, Hubbard AE, Steel C, Kubofcik J, Hamlin KL, et al. (2017) https://doi.org/10.1371/journal.pntd.0005616

28. File downloads
Forks
See the Impact

29. NEXT STEPS
1.Build a test project on the OSF
2.Document from the beginning - or even
right now
3.Talk to your collaborators
– What is our data management plan?
– What/when will we share?

30. Jennifer Freeman Smith, PhD
Center for Open Science
Charlottesville, VA, USA
jennifer@cos.io
@jfsmith434
Find this presentation at https://osf.io/ncdpa/
QUESTIONS AND COMMENTS

31. SUPPLEMENTAL SLIDES

32. The
Research
Lifecycle

33. POSITIVE
RESULTS BY
DISCIPLINE
Fanelli D (2010) “Positive”
Results Increase Down the
Hierarchy of the Sciences.
PLOS ONE 5(4): e10068.
doi:10.1371/journal.pone.001
0068
http://journals.plos.org/ploso
ne/article?id=10.1371/journal
.pone.0010068

34. RESEARCHER DEGREES OF
FREEDOM
All data processing and analytical choices made
after seeing and interacting with your data
• Should I collect more data?
• Which observations should I exclude?
• Which conditions should I compare?
• What should be my main DV?
• Should I look for an interaction effect?

35. FALSE POSITIVE INFLATION
Simmons, Nelson, & Simonsohn (2011)

36. EXPLORATORY VS.
CONFIRMATORY ANALYSES
Exploratory
• Interested in exploring possible
patterns/relationships in data to develop hypotheses
Confirmatory
• Have a specific hypothesis you want to test
Preregistered analysis plans clarify which results are
exploratory and which are confirmatory

37. Other ways the OSF
supports open workflows

38. Preprints decouple
publication and
evaluation, allowing for the
rapid dissemination of content.

39. OSF
Preprints

41. Free, open
(meta)dataset of
research activity across
the research lifecycle
~40M records from
~163 sources
http://share.osf.io

43. PRE-REGISTRATION
Documenting your research plan in a read-only
public repository before you conduct the study.
Pre-registration helps reduce the “file drawer effect”
by increasing discoverability of unpublished studies.

44. PRE-REGISTRATION
Benefits of pre-registering your study depend on
how much information you include. At a minimum
a preregistration should include the “what” of the
study:
• Research question
• Population and sample size
• General design
• Variables you’ll be collecting, or dataset you’ll be
using

45. PRE-ANALYSIS PLAN
Details the analyses planned for hypothesis testing:
Sample size
Data processing and cleaning procedures
Exclusion criteria
Statistical analyses
Including a pre-analysis plan in your pre-registration helps
improve study accuracy and replicability by guarding
against unintended false positive inflation.

46. $1,000,000 PREREGISTRATION
CHALLENGE

47. $1,000,000 PREREGISTRATION
CHALLENGE
https://cos.io/prereg