1. Open Access Article Quality: Error
Rates in References as an Indicator
Joseph R. Kraus
Penrose Library, University of Denver
@jokrausdu
Presentation for the Library 2.012 Conference
October 4, 2012

2. Introduction
• There are some individuals and publishers who
claim that Open Access journals are lower quality
compared to traditionally published journals.
– “PLoS is teetering on the edge of becoming viewed as
a low‐quality, high‐volume publisher — a far cry from
the promise the initiative once held.” (Kent Anderson,
Scholarly Kitchen, April 27, 2010)
– There is a “perception that open‐access journals are
of lower quality than traditional publications.” (Josh
Fischman, Chronicle of Higher Education, February 8,
2011)

3. Journal Quality vs Article Quality
• The impact factor is often touted as an
indicator of journal quality. That number
provides the average number of citations to
articles in the journal over a two year span.
• It is difficult to evaluate the quality of
individual journal articles. The presenter
considers another method to determine a
quality of individual articles.

4. The study
• Compare the error rates from the reference
sections of four Open Access articles (PLOS
ONE) with four articles from a commercial
publisher’s journal (Stem Cell Research from
Elsevier).
• This could be considered a method to
determine the carefulness of authors, and
hence, a quality indicator of individual articles.

5. Background
• In earlier research, Mandy Taha and I found
that reference section error rates were similar
between a couple of Open Access journal
articles and many commercial‐based journal
articles.
– 2 articles were OA while another 13 articles were
from a two different commercial publishers
(Elsevier and Springer)
– This was a not the focus of the research at the
time.

6. Article Type of publication Total Number of Errors Error rate
Number References
1 OA 30 2 6.7%
2 Commercial 27 2 7.4%
3 Commercial 37 1 2.7%
4 Commercial 30 1 3.3%
5 Commercial 56 4 7.1%
6 OA 46 8 17.4%
7 Commercial 41 2 4.9%
8 Commercial 68 0 0.0%
9 Commercial 70 1 1.4%
10 Commercial 32 1 3.1%
11 Commercial 58 6 10.3%
12 Commercial 32 1 3.1%
13 Commercial 23 0 0.0%
14 Commercial 35 8 22.9%
15 Commercial 56 4 7.1%

7. Other research on error rates
• Article Reference Accuracy in Peer‐Reviewed Pediatric
Orthopaedic Literature (26%)
• Accuracy of references in the ophthalmic literature (16%)
• Accuracy of references in general surgical journals — An old
problem revisited (11.1%)
• The accuracy of references in paediatric journals (29.7%)
• Citation and quotation accuracy in three anatomy journals
(27%)
• The accuracy of citation and quotation in
otolaryngology/head and neck surgery journals (37.5%)
• Reference List Accuracy in Social Work Journals (41.2%)

8. What I evaluated?
• 4 recent OA articles in the field of stem cells
from the journal, PLOS ONE
• The IF of PLOS ONE is 4.092 (12th out of 84 in
Biology)
• 4 recent articles from the journal Stem Cell
Research (published commercially by Elsevier)
• The IF of Stem Cell Research is 5.127 (48th out
of 180 in Cell Biology)

9. PLOS ONE articles
• Dopaminergic Neurons from Midbrain‐Specified Human
Embryonic Stem Cell‐Derived Neural Stem Cells Engrafted
in a Monkey Model of Parkinson’s Disease (53 references)
• Generation of Induced Pluripotent Stem Cells from the
Prairie Vole (56 references)
• Amniotic Mesenchymal Stem Cells Enhance Wound Healing
in Diabetic NOD/SCID Mice through High Angiogenic and
Engraftment Capabilities (45 references)
• Pleiotropy of Glycogen Synthase Kinase‐3 Inhibition by
CHIR99021 Promotes Self‐Renewal of Embryonic Stem Cells
from Refractory Mouse Strains (54 references)
• 208 reference total

10. Stem Cell Research articles
• Sonic hedgehog released from scratch‐injured
astrocytes is a key signal necessary but not sufficient
for the astrocyte de‐differentiation (38 references)
• A novel variant of Oct3/4 gene in mouse embryonic
stem cells (40 references)
• DUXO, a novel double homeobox transcription factor, is
a regulator of the gastrula organizer in human
embryonic stem cells (34 references)
• Proton‐gated ion channels in mouse bone marrow
stromal cells (41 references)
• 153 references total

11. Results
• PLOS ONE, 7 errors within 208 references (3.4%)
– Volume and page information
– Book editor, EYS instead of EY Snyder
– Author, Kim KS instead of Kim K‐S
– Author, Bergstorm instead of Bergstrom
– Only one author, et al., instead of listing the first five
authors and then et al.
– No year
– Author, Nutter LM instead of Nutter LMJ

12. Results
• Stem Cell Research, 9 errors within 153
references (5.9%)
– 3 wrong Last names, Kaya should be Sahin Kaya;
Altaba should be Ruiz i Altaba; Peter, H. should be
Holzer, P.
– 3 Wrong page numbers
– Wrong volume number
– Missing author
– Wrong article number

13. Findings
• Authors who write for PLOS ONE seem to be
just as careful with formatting their references
as authors who write for the Elsevier journal,
maybe even more so.

14. Items not considered errors or typos
• PLoS One close enough to PLOS ONE
• Brustle instead of Brüstle
• Munoz‐Sanjuan instead of Muñoz‐Sanjuán
• In a cited paper, the first author is listed as Da
Wei Huang. PubMed lists the author as Huang
da W. But, where should the comma go in the
reference list? Should it be Huang, da W. or
Huang da, W.?

15. Notes
• For the PLOS ONE articles, I could not use the Web of
Science to show me which ones might have incorrect
information
– The reference with Bergstorm went to the correct
reference with Bergstrom.
• For the Stem Cell Research articles, I could use the
Scopus system to indicate which references might be
incorrect.
– If I saw that a reference was cited 0‐3 times in Scopus, I
would double check the information in that reference.
– There could be even more errors in these four Stem Cell
Research articles than what I initially found.

16. For example

17. Another example

18. More notes
• I may not have caught all of the errors concerning
hyphenated first names. (Smith AB should have
been listed at Smith A‐B)
• PLOS Authors correctly cited others that had
difficult to spell last names, such as
Bandyopadhyay
• PLOS Authors correctly kept British words in titles
(behaviour)
• PLOS Authors correctly cited this Lancet article.

20. Questions for the audience, 1 of 2
• Should I expand this research to include:
– More articles from PLOS?
– Articles from other OA publishers such as BMC or
Hindawi or others?
– Articles from Springer or Wiley or others?

21. Questions for the audience, 2 of 2
• What are other ways to indicate article
quality?
– Numbers of citations via WOS and Google Scholar.
Been there, done that.

22. Questions for me?