O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Liz Allen - Open peer review (through the lens of F1000’s open research publishing platforms)

55 visualizações

Publicada em

With the progress towards open science, scientific communication is facing a new wave of innovations towards more openness and speed of research publication which will deeply affect the way the peer review function is carried out and the overall role of journals in assuring quality and adding value to manuscripts.

Several initiatives are promoting the generalized adoption of open access preprints as a formal beginning stage of research publication, which has been common since the 90’s in the physics community. And, in the last decade, new ways to carry out the evaluation of manuscripts have emerged either to replace or to improve the traditional methods, which are widely criticized as being slow and expensive in addition to lacking transparency.

Quality nonprofit journals from emerging and developing countries have succeeded to follow the main innovations brought by the Internet. In addition to the technicalities of the digital publishing, there is a wide adoption of Open Access in the international flow of scientific information. The new wave of innovations that affect the peer review function and the changing role of journals pose new challenges to the emerging and developing countries in regard of scientific publishing. The adoption of these innovations is essential for progress of SciELO as a leading open access program to enhance scientific communication.

The scope of this workshop aims at an in-depth analysis and discussion of the state of art and main trends of the peer review function, the modalities of carrying it out as well as of the increasing adoption of mechanisms to speed publication such as preprints and how they affect and potentially renew the role of journals. These recommendations will guide SciELO policies on manuscript evaluation and on the adoption of preprint publications.

Publicada em: Ciências
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

Liz Allen - Open peer review (through the lens of F1000’s open research publishing platforms)

  1. 1. Open peer review Liz Allen Director of Strategic Initiatives, F1000 Scielo 20| Sao Paolo| September 2018 (through the lens of F1000’s open research publishing platforms) @allen_liz
  2. 2. F1000’s Director of Strategic Initiatives (2015 – present) Head of Evaluation at Wellcome (2000 - 2015 ) ORCID Board Director (2010 – 2015) Co-led development of project CRediT (2010 - present) Crossref Board Director (2017 – present) Visiting Senior Research Fellows at Policy Institute @ Kings College London Love all things research outputs, metrics & ‘science of science’ About me (declarations)
  3. 3. Introductory thoughts A crisis in peer review? The future is open … The case of F1000’s post-publication open peer review model Exploring the benefits and challenges of open peer review Thoughts? Discussion? Outline of discussion
  4. 4. Source: https://www.editage.com/insights/7-common-types-of-academic-peer-reviewwitter
  5. 5. A crisis in peer review?
  6. 6. Source: https://www.forbes.com/sites/geoffreykabat/2015/11/23/the-crisis-of-peer-review/#61e5474c463e “If peer review was a drug, it would never get on the market." Geoffrey Kabat Author & Cancer Epidemiologist (quoted in 2015)
  7. 7. Hannes Alfvén (1908-1995) awarded Nobel Prize for physics 1970 "The peer review system is satisfactory during quiescent times, but not during a revolution in a discipline such as astrophysics, when the establishment seeks to preserve the status quo."
  8. 8. Source: https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 Source: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(13)62329-6/
  9. 9. “There are many details to be resolved, but the basic principle of independent, expert peer review seems to be at the absolute core of scientific [publishing] Source: Royal Society (2015) Future of Scholarly Scientific Communication https://royalsociety.org/~/media/events/2015/04/FSSC1/FSSC-Report.pdf Royal Society (2015) … we need to retain that basic principle, however we go about organising it.”
  10. 10. “Although frequently criticized, peer review still plays an important role in validating research results and advancing discovery. Source: Digital Science (2016) What might peer review look like in 2030? https://www.digital-science.com/blog/news/the-future-of-peer-review-new-report-by-biomed-central-and-digital-science- spotonreport/ We want to start conversations with all stakeholders to find progressive ways of improving peer review for researchers globally and across all disciplines.”
  11. 11.  Tailoring & selection precision  Diversity  Experimentation – new approaches  Training & mentoring  Cross-publisher sharing /portability/ efficiencies  Recognition, credit & reward for reviewers  Technology to improve effectiveness Recommendations Source: Digital Science (2016) What might peer review look like in 2030? https://www.digital-science.com/blog/news/the-future-of-peer-review-new-report-by-biomed-central-and-digital-science- spotonreport/
  12. 12.  Tailoring & selection precision  Diversity  Experimentation – new approaches  Training & mentoring  Cross-publisher sharing /portability/ efficiencies  Recognition, credit & reward for reviewers  Technology to improve effectiveness Recommendations Source: Digital Science (2016) What might peer review look like in 2030? https://www.digital-science.com/blog/news/the-future-of-peer-review-new-report-by-biomed-central-and-digital-science- spotonreport/
  13. 13. Source: https://www.editage.com/insights/7-common-types-of-academic-peer-review#.Wb8ag6Uf074.twitter
  14. 14. The future is open …
  15. 15. Source: https://www.fosteropenscience.eu/content/what-open-science-introduction
  16. 16. https://sfdora.org
  17. 17. Ross-Hellauer T. What is open peer review? A systematic review F1000Research 2017, 6:588 (doi: 10.12688/f1000research.11369.1) “ … open peer review (OPR) as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the ethos of Open Science …
  18. 18. ‘Taxonomy’ of open peer review Ross-Hellauer T. What is open peer review? A systematic review F1000Research 2017, 6:588 (doi: 10.12688/f1000research.11369.1)
  19. 19. ‘Open peer review (OPR) is a cornerstone of the emergent open science agenda. Yet to date no large scale survey of attitudes towards OPR amongst academic editors, authors, reviewers and publishers.’ Survey (2016) sample n=3000+ Ross-Hellauer T, Deppe A, Schmidt B (2017) Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLOS ONE 12(12): e0189311. https://doi.org/10.1371/journal.pone.0189311 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311
  20. 20. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311
  21. 21. Ross-Hellauer T, Deppe A, Schmidt B (2017) Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLOS ONE 12(12): e0189311. https://doi.org/10.1371/journal.pone.0189311 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311
  22. 22. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311
  23. 23. HOW WE SHARE & TALK ABOUT SCIENCE IS CHANGING FAST
  24. 24. Widening range of research outputs available Data Resources Software all with a trackable & persistent DOIs
  25. 25. Source: 2017 http://asapbio.org/ Growth in rapid publication models
  26. 26. Evolving policy & research funding landscape
  27. 27. Source: https://www.nature.com/articles/d41586-018-06178-7
  28. 28. Introducing F1000’s publishing model
  29. 29. Publish. Engage. Accelerate. • Post-publication peer review • Open peer review • Open data • Open access • Versioning • Indexing • Comprehensive meta-data • Full usage metrics • Living figures • more to come ….
  30. 30. F1000’s post-publication, open peer review Preprint-like stage
  31. 31. F1000’s post-publication, open peer review  invited peer review  indexing in bibliographic databasesPreprint-like stage
  32. 32. 1 ………………… 2 ………………… 3 ………………… 4 ………………… 5 ………………… Author Author suggestions AI tool Memory Editor Editorial verificationAuthor selection F1000 peer review selection model Authors control the selection of reviewers suggested for peer review
  33. 33. ‘published’ & awaiting peer review Article immediately citable with a DOI and usage indicators Article status noted: awaiting peer review Article version clear ...
  34. 34. ‘published’ & undergoing peer review
  35. 35. http://cartoonsbyjosh.co.uk
  36. 36. Source: https://www.editage.com/insights/7-common-types-of-academic-peer-review#.Wb8ag6Uf074.twitter
  37. 37. “ … undoubtedly an increase in the use of researchers publishing their research on alternative platforms for biomedical sciences …” Kirkham J and Moher D. Who and why do researchers opt to publish in post-publication peer review platforms? - findings from a review and survey of F1000 Research F1000Research 2018, 7:920
  38. 38. Kirkham J and Moher D. Who and why do researchers opt to publish in post-publication peer review platforms? - findings from a review and survey of F1000 Research F1000Research 2018, 7:920 Motivation to publish on F1000Research: open access? 84%
  39. 39. Kirkham J and Moher D. Who and why do researchers opt to publish in post-publication peer review platforms? - findings from a review and survey of F1000 Research F1000Research 2018, 7:920 Motivation to publish on F1000Research: open peer review? 74%
  40. 40. Kirkham J and Moher D. Who and why do researchers opt to publish in post-publication peer review platforms? - findings from a review and survey of F1000 Research F1000Research 2018, 7:920 Motivation to publish on F1000Research: speed of publication? 81%
  41. 41. Easier adoption in certain research areas?
  42. 42. Publication trends F1000Research
  43. 43. Debating the benefits & challenges of OPR
  44. 44. Response rates in open system? Variation by demographics seniority? geography? etc Level and nature of reviews in open system: detail & nature of response (e.g. constructive?) Levels of agreement & tendency towards group think? Time & resource burden: open vs closed models Ability to get recognition and credit for peer review work System efficiencies? Reduce resource involved in finding & securing reviews Outline of discussion: OPR experimentation
  45. 45. Response rates in open system? Variation by demographics seniority? geography? etc Level and nature of reviews in open system: detail & nature of response (e.g. constructive?) Levels of agreement & tendency towards group think? Time & resource burden: open vs closed models Ability to get recognition and credit for peer review work System efficiencies? Reduce resource involved in finding & securing reviews Outline of discussion: OPR experimentation
  46. 46. Levels of agreement between reviewers in open, post-publication model Results 1,133 articles = 2,266 reviews Median time between first two reviews = 18 days [interquartile 6-52 days] Cohen K indicated only a ‘fair agreement’ between reviewers; changing minimally over time between review Evidence? Agreement between reviewers did not change over time No evidence that reviewers are systematically influenced by seeing the review of a previous reviewer But Need comparators Monitor over time as new, open models become more commonplace Source: https://f1000research.com/posters/6-1678
  47. 47. ‘published’ & undergoing peer review Transparent review history
  48. 48. Response rates in open system? Variation by demographics seniority? geography? etc Level and nature of reviews in open system: detail & nature of response (e.g. constructive?) Levels of agreement & tendency towards group think? Time & resource burden: open vs closed models Ability to get recognition and credit for peer review work System efficiencies? Reduce resource involved in finding & securing reviews Outline of discussion: OPR experimentation
  49. 49. Review activity with trackable & persistent DOIs & linked to ORCID & Crossref metadata
  50. 50. … near ORCID id full house!
  51. 51. Recognition for peer review efforts
  52. 52. Response rates in open system? Variation by demographics? seniority? geography? etc Level and nature of reviews in open system: detail & nature of response (e.g. constructive?) Levels of agreement & tendency towards group think? Time & resource burden: open vs closed models Ability to get recognition and credit for peer review work System efficiencies? Reduce resource involved in finding & securing reviews Outline of discussion: OPR experimentation
  53. 53. Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2
  54. 54. Gender of submitting authors 58 Gender of referees Total submitting authors: 258 Total suggested referees: 1908 2% 47%51% 6% 31% 63% Female Male Unknown Overall gender balance on WOR Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2
  55. 55. 59 Gender balance of author-suggested referees Gender of author-suggested referees Gender of referees suggested by a female author Total referees suggested by authors: 1545 Total referees suggested by female authors: 749 Female Male Unknown 6% 31% 63% 4% 60% 36% Gender of referees suggested by a male author Total referees suggested by male authors: 743 6% 27% 67% Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2
  56. 56. Gender balance of AI and editor suggested referees 60 5% 33%62% Gender of editor-suggested referees Gender of referees suggested by the algorithm Total referees suggested by editors: 183 Total referees suggested by the tool: 329 6% 31% 63% Female Male Unknown Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2
  57. 57. Key findings: • Author selected peer review provides comparable gender balance to other publisher studies (~25-35% female reviewers). • There is little difference in gender balance when comparing author suggestions, editorial suggestions and algorithmic suggestions. • Male authors suggest a higher % of male reviewers, while female authors suggest a higher % of female reviewers – comparable to other publisher studies. Source: https://figshare.com/articles/Diversity_and_inclusion_in_peer_review_and_beyond/7098674/2
  58. 58. What do you think? Liz Allen Director of Strategic Initiatives, F1000 Scielo 20| Sao Paolo| September 2018 @allen_liz
  59. 59. Right click background and select Format Background Upload your background image Step 1 Step 2
  60. 60. Title option 3 - Photographic Author name

×