Brief presentation for University of Minnesota Senate Library Committee, February 10, 2016.

1. Beyond the Factor:
Talking about research impact
Claire Stewart
Associate University Librarian for Research & Learning
Senate Library Committee, February 10, 2016

2. Interest in metrics
• In hiring, in support of promotion & tenure
• Funders and publishers, evaluating proposals
• Institutional productivity
• Impact on our communities

3. Abbreviated metrics overview
Alphabet Miso. https://www.flickr.com/photos/bean/322616749/

4. JIF: Journal Impact Factor
Source: The Thompson Reuters Impact Factor
Significant variance across
disciplines:
• Top ranked journal overall:
JIF = 144.800
• Top ranked journal in history:
JIF = 2.615
Not based on any single
author/article
Oft criticized (DORA, Leiden,
HEFCE statements)

5. eigenfactor
Based on same citation source as the Impact Factor (Thomson’s Journal
Citation Reports)
Weights journals by importance based on citation frequency, similar to
Google page rank
Also calculates an Article Influence score, over the first 5 years of an article

6. h index
Scholar-specific:
“A scientist has index h if h of his or her Np papers have
at least h citations each and the other (Np – h) papers
have ≤h citations each.”
Dependent on citation index source (Google scholar
and Scopus might have different values)
Doesn’t really account for different citation/usage
patterns between fields
Source: Hirsch, J. E. “An Index to Quantify an Individual’s Scientific Research Output.” Proceedings of the
National Academy of Sciences of the United States of America 102, no. 46 (November 15, 2005): 16569–72.
doi:10.1073/pnas.0507655102.

7. Altmetrics
• Shares and mentions in non-traditional places,
on social media, etc. (twitter, FB, Mendeley,
blogs, wikipedia etc.)
• Often dependent on identifiers (DOIs,
PubMedIDs, arXivIDs, etc.) which can have
lower penetration in arts & humanities fields

9. Article level metrics

10. Recent expressions of concern about
strictly quantitative approach to research
assessment

11. Advice to HEFCE (UK)
Framework for responsible
metrics
• Robustness: use the best
possible data
• Humility: quantitative should
support expert assessment
(e.g., peer review)
• Transparency: be able to show
where data came from & let
results be verified
• Diversity: account for
variation by field
• Reflexivity: indicators updated
as the system & effects change
20 specific recommendations to
HEFCE around use of metrics

12. What do we want to know when we
talk about impact?
• How has this [researcher’s] work advanced
knowledge?
• Has this research been evaluated and by whom?
• What is field shaping research?
• Who are the researchers shaping my field?
• What is going on in my field that’s important, or
in a field that could benefit my work?
• What is the broader societal benefit of this work?
(value of higher education, research investments)

14. Source: Weinberg, Bruce A., Jason Owen-Smith, Rebecca F. Rosen, Lou Schwarz,
Barbara McFadden Allen, Roy E. Weiss, and Julia Lane. “Science Funding and Short-
Term Economic Activity.” Science 344, no. 6179 (April 4, 2014): 41–43.
doi:10.1126/science.1250055.
Where was CIC federal research funding $ actually spent?

15. IRIS sample products, November 2015
Grad students, postdocs and other research staff employment

17. What are the other questions we will
want to ask?
And what kinds of information will we need to
answer these questions?

18. What are the other questions we will
want to ask?
Who at UMN is doing research in or about
countries other than the United States? Who are
they collaborating with? What kind of effect has
this work had?
‘Effect’ could include: articles, books and reports published,
presentations offered, information about who benefited from
these outputs, integration into policy development
(conversations about and/or new legislation, regulation, etc.)

19. Why is this hard?
• Wide variety in what constitutes a valuable
research output/indicator across disciplines
• Types of outputs expanding
• Data about research outputs is messy partly
because it has the typical big data problems:
volume, velocity, variety
• Highly distributed scholarly communication
infrastructure (the data about outputs is
everywhere)

20. Outputs/indicators vs metrics
“The observations here relate to the fact that while there
is unease about the use of metrics as a mode of
‘measuring’ the excellence of research produced in the
UK’s HEIs, the rich array of data presented as part of
REF2014 demonstrates that the arts and humanities
sector are comfortable with deploying numbers (albeit
framed as data rather than metrics) to present a case
about the excellence of their research cultures.”
Mike Thelwall, and Maria M Delgado. “Arts and Humanities Research Evaluation: No Metrics Please,
Just Data.” Journal of Documentation 71, no. 4 (June 25, 2015): 817–33. doi:10.1108/JD-02-2015-
0028.

21. Outputs/indicators vs metrics

22. Discussion
What conversations are taking place in your field
about impact and metrics? What outputs are of
interest? Do we know how to describe them?
Are they captured in any consistent way?