Altmetrics is an emerging area encompassing broader assessment of scholarly impact through downloads, links, and online conversations to fill gaps in assessing research. Bibliometrics is the traditional form of measuring the impact of scholarly research through citation rates. The Research & Instruction Librarian for Sciences and the Scholarly Communication Librarian at Wake Forest University will compare bibliometrics and altmetrics, and discuss their applications in science information literacy and research assessment in higher education.
The Impact Factor, Eigenfactor, and Altmetrics: From Theory to Analysis
1. +
The Impact
Factor, Eigenfa
ctor, &
Altmetrics:
From Theory to Analysis
Sarah Jeong, Research & Instruction Librarian for Science
Molly Keener, Scholarly Communication Librarian
Wake Forest University
October 18, 2013
5. +
“In the early 1960s Irving H.
Sher and [Eugene Garfield]
created the journal impact
factor to help select journals
for the Science Citation
Index…[Garfield] expected
that it would be used
constructively while
recognizing that in the wrong
hands it might be abused.”
(Garfield 1999)
6. +
Eugene Garfield
(1955) first mentioned
the concepts of
Science Citation Index
and Impact Factor in
Science
Impact Factor (IF) = “a measure of the
frequency with which an „average article‟
in a journal has been cited in a particular
year or period” wokinfo.com/essays/impact-factor/
2005 IF of a journal =
2005 cites to articles published in 2003-04
number of articles published in 2003-04
“The journal impact factor is a
good predictor of the quality of
journals as measured by
citations to primary research
articles. It is, however, a poor
indicator of citations to specific
papers or of the future
performance of individual
researchers.”
(Nature Materials 2013)
7. +
PROS
Impact Factor
•
•
•
•
One of the oldest quantified metrics
2-year & 5-year citation windows
Many journals advertise their IF
Widely used & recognized
8. +
Impact Factor
CONS
• Citations need context
•
•
•
Unidentifiable + or – citations
Self-citations
Review articles are favored
• Metric for journals not authors
• Time varying IF
• Limited to JCR
9. +
“For the few scientists who
earn a Nobel Prize, the
impact…of their research is
unquestionable. For the rest
of us, how does one
quantify the cumulative
impact…of an individual‟s
scientific research output?”
(Hirsch 2005)
10. +
h-index shows the broad impact of
an individual’s work
h-index developed
by a physicist
(Hirsch 2005)
Ex. Dr. Kim’s h-index = 12
12 of his articles have been cited at
least 12 times each
11. +
PRO
Considers the impacts of both
journals and authors
CONS
h-index
Unidentifiable + or – citations
h-index increases with age so
comparing productivity of younger
researchers is problematic
Calculated in Web of Science but
need comprehensive citation
report of all author’s publications
12. +
Dr. Bergstrom and his
colleagues “have developed
a way to use the network
structure of citations to
improve on simple citation
counts in measuring the
scientific influence of
academic publications.”
(Bergstrom 2007)
13. +
Eigenfactor developed
by Dr. West and
Dr. Bergstrom at
Univ. of Washington
“Eigenfactor
scores are
scaled so that the sum of
the Eigenfactor scores of all
journals listed in Thomson's
Journal Citation Reports
(JCR) is 100.”
eigenfactor.org
14. +
PROS
Weighted metric with different
weights for journals
Excludes self-citations
5-year citation window
Eigenfactor
CONS
Limited to journals in Journal
Citation Reports
Journals assigned to a single
subject category (Jacso 2012)
17. +
Altmetrics measure…
How far and wide content travels through the scholarly* web
Web-driven social scholarly interactions
Twitter
Facebook
Blogging
Bookmarking
* It‟s not just scholars who are engaging:
clinicians, practitioners, and the general public are reading and
sharing, too!
18. +
Altmetrics vs. Article-Level Metrics
Related, but not interchangeable
Article-Level Metrics present picture of an article’s true impact
via data points
Scopus
Nature
PLoS
Highwire
Altmetrics track other types of output in addition to
articles, including datasets, presentations, and software
23. +
So what’s the verdict?
PROS
New and emerging
Gaming still possible
Content-level, not containerlevel
CONS
Context is critical
Immediacy
Social sharing
Incorporates traditional metric
measures, too
29. +
Assessing “impact”
Tenure & Promotion
Expectations
Tenure-track vs. tenured
New models & modes of scholarship
Digital Humanities
San Francisco Declaration on Research Assessment (DORA)
34. +
References - Bibliometrics
Bergstrom, C. (2007). Eigenfactor: Measuring the value and prestige of scholarly journals. College & Research
Libraries News, 68(5), 314–316.
Beware the impact factor. (2013). Nature Materials, 12(2), 89–89. doi:10.1038/nmat3566
Eigenfactor Score and Article Influence Score: Detailed Methods. (2008). Retrieved from
http://www.eigenfactor.org/methods.pdf
Garfield, E. (1955). Citation indexes for science. Science, 122(3159), 108–111.
Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979–980.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National
Academy of Sciences of the United States of America, 102(46), 16569–16572. doi:10.1073/pnas.0507655102
Jacsó, P. (2010). Eigenfactor and article influence scores in the Journal Citation Reports. Online Information
Review, 34(2), 339–348. doi:http://dx.doi.org/10.1108/14684521011037034
Jacsó, P. (2012). The problems with the subject categories schema in the EigenFactor database from the
perspective of ranking journals by their prestige and impact. Online Information Review, 36(5), 758–766.
doi:10.1108/14684521211276064
Pritchard, A. (1969). Statistical bibliography or bibliometrics? Journal of Documentation, 25(4), 348–349.
Van Noorden, R. (2013). Brazilian citation scheme outed. Nature, 500(7464), 510–511. doi:10.1038/500510a
35. +
References - Altmetrics
Altmetric.com
Altmetrics.org
DORA – am.ascb.org/dora
Galligan, F., & Dyas-Correia, S. (2013). Altmetrics: Rethinking the Way We Measure. Serials Review, 39, 56-61. doi:
10.1016/j.serrev.2013.01.003
Galloway, L.M., Pease, J.L., & Rauh, A.E. (2013). Introduction to Altmetrics for Science, Technology, Engineering, and
Mathematics (STEM) Librarians. Science & Technology Libraries, epub ahead of print, 11pp. doi:
10.1080/0194262X.2013.829762
ImpactStory.org
Kwok, R. (2013). Research Impact: Altmetrics Make Their Mark. Nature, 500, 491-493. doi: 10.1038/nj7463-491a
Lapinski, S., Piwowar, H., & Priem, J. (2013). Riding the Crest of the Altmetrics Wave. College & Research Libraries
News, 74(6), 292-300.
Plumanalytics.com
Tananbaum, G. (2013). Article-Level Metrics: A SPARC Primer. SPARC. 14pp.
Now is classified by the National Library of Medicine MeSH broader term“Bibliometrics” Narrower Term “Impact Factor”
Each journal can belong in multiple subject categories
Variation of IF range among disciplines Ex. medical v. mathematics journals
Dr. Kim has written 39 articles so his maximum h-index could possibly equal 39. According to a citation report in Web of Science database, Dr. Kim’s h-index is 12, therefore 12 of his articles have been cited at least 12 times each.
Eigenfactorconsiders citations from more influential journals than lesser known journals
Users may not guess correctly to which subject category the journal was assigned.
As scholars have migrated to online spaces for research purposes, various platforms capture previously invisible activities such as reading, saving, discussing, and recommending research outputsTraces of these activities are being observed to “inform new metrics of scholarly influence and impact—so called altmetrics” (Lapinski, Piwowar, Priem)Short for alternative metrics, Jason Priem coined the term, and in “Altmetrics: A Manifesto” he co-authored, he fully proposed altmetrics in 2010; defined as “altmetrics is the creation and study of new metrics based on the Social Web for analyzing, and informing scholarship.”
“Aims to measure Web-driven scholarly interactions” (Howard, quoted in Galligan & Dyas-Correia)“Altmetrics are new measurements for the impact of scholarly content, based on how far and wide it travels” (Galligan, quoted in Galligan & Dyas-Correia)“Altmetrics go beyond traditional citation-based indicators as well as raw usage factors (such as downloads or click-through rates) in that they focus on readership, diffusion and reuse indicators”“More importantly, they are diverse, tracking impacts all across a quickly changing landscape populated by: diverse products…; diverse platforms…; diverse audiences…” (Lapinski, Piwowar, Priem)Measures: usage, captures, mentions, social media, citations
“Sometimes conflated, but not interchangeable” (Tananbaum)Public Library of Science, also known as PLoS; Scopus; Nature Publishing Group; and Highware have all incorporated article-level metrics into their platformsUnlike the bibliometrics measures Sarah discussed, which are usually tied to a journal’s impact, ALMs “disaggregate an individual article’s impact from the publication in which it appears” (Tananbaum)Another way to think about this is that ALMs track the impact of a single article and altmetrics, when aggregated, track the impact of multiple research components, from publications to data to presentations to products
ImpactStory is one tool for tracking altmetricsHeather Piwowar and Jason Priem cofounded; recently received a huge NSF grant (300K) and also is funded by the Alfred P. Sloane FoundationOpen source altmetrics tool; nonprofit companyDraws from Facebook, Twitter, CiteULike, Delicious, PubMed, Scopus, CrossRef, scienceseeker, Mendeley, Wikipedia, slideshare, Dryad, and figshareNormalizes metrics based on a sample of articles published the same yearRaw and percentiles compared to other articles
Plum Analytics is another tool for tracking altmetricsProprietaryMarketed to universities and research institutions, not directly to researchersMetrics for nearly two dozen types of outputsCustom reports intended to quantify departmental productivity, support grant proposals, and address other impact-related questions
Altmetric.comProprietaryAdopted by publishers, including Springer, Nature Publishing Group, Scopus (Elsevier), and BioMed CentralTracks social media sites, newspapers, and magazines for mentions of scholarly articlesScore created for each article; quantitative measure of the quality and quantity of attention an article has receivedBased on three main factors: number of individuals mentioning the paper, where the mentions occurred, and how often the author of each mention talks about scholarly articles
PLOS ONE article-level metrics
Since I am the library liaison to the Biology, Chemistry, and Physics departments, I compared the top ranked scientific journals by 2-year Impact Factor in Journal Citation Reports.
DORA drafted in December 2012The San Francisco Declaration on Research Assessment (DORA), initiated by the American Society for Cell Biology (ASCB) together with a group of editors and publishers of scholarly journals, recognizes the need to improve the ways in which the outputs of scientific research are evaluated. The group met in December 2012 during the ASCB Annual Meeting in San Francisco and subsequently circulated a draft declaration among various stakeholders. DORA as it now stands has benefited from input by many of the original signers listed below. It is a worldwide initiative covering all scholarly disciplines. We encourage individuals and organizations who are concerned about the appropriate assessment of scientific research to sign DORA.A number of themes run through these recommendations:the need to eliminate the use of journal-based metrics, such as Journal Impact Factors, in funding, appointment, and promotion considerations;the need to assess research on its own merits rather than on the basis of the journal in which the research is published; andthe need to capitalize on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact).
DryadCurated general-purpose repository for data underlying scientific and medical literatureIntegrated data submission for a growing number of journalsIndividual and institutional membership options available$80/deposit fee for individualsData submitted under a CC Zero license; given DOIsRuns on Dspace, served by NC State UniversityFigshareRepository for figures, datasets, media, papers, posters, presentations, and filesets; given DOIsSupports multiple formatsPrivate and public spaceCollaborative spaces in the worksDatasets shared under a CC Zero license; everything else under a CC-BY licensePreserved in CLOCKSSFree to join; funded by Digital ScienceGitHubworld’s largest open source communityCollaborative project spacePublic and private options available
ZoteroOpen source, cloud based citation management systemMore easily facilitates sharing of citationsMendeleyWas independent, recently acquired by Elsevier, so we’ll see what happensEnables full text article uploadsMore social aspect than Zotero, but also a citation management systemF1000For biology and medicineOA journal and OA database of posters/presentationsRecommended top articles by top faculty and researchers worldwideCiteULike- Yet another social citation management tool
Academia.eduOver 4.8 million usersOver 1.6 million papers and 888,000+ research interests addedShare papers, see analytics on papers and profile, follow others in your fieldResearchGate“built by scientists, for scientsts”Over 4 million membersShare papers, find collaborators, track stats (views, downloads, citations of your work)SlideShareNot as overtly geared toward research, but used by many researchersMarketing angleCan follow othersCan get usage stats on views, downloadsCan upgrade to Pro account for additional benefits and features