This document summarizes the results of a survey of over 3,000 researchers about their awareness and perceptions of different metrics for measuring research quality and impact. The key findings are: - The Impact Factor was the most widely known metric (82%), followed by the h-index (43%) and Journal Usage Factor (10%). Altmetrics were the least known (1%). - Impact Factor (64%), h-index (58%) and citations (29%) were most often seen as the most useful metrics for measuring research quality. - Generally, metrics with higher awareness tended to also be considered more useful. However, the h-index and citations were seen as more useful than typical given their awareness levels.

1. 5 years ago…
Nearly 50% of respondents saw Web 2.0 playing a key role
in “Providing Quality Indicators” in 5 years (2013)
Source: (2collab) Social Media survey - May 2008 - 1,824 respondents

2. 1 year ago… October 2012
Background & approach:
54,442 individuals were randomly selected from
Scopus
3,090 respondents completed
Representative response by country and discipline.
Error margin 1.5%, at 90% confidence levels
2
Adrian Mulligan, Gemma Deakin and Rebekah Dutton
Elsevier Research & Academic Relations

3. 3
Most widely known by researchers
Impact Factor 82%
H-Index 43%
Journal Usage Factor 10%
????????? 1%
Impact Factor is published by Thomson Reuters, Altmetrics were least well known

4. 4
Impact Factor 82%
H-Index 43%
Journal Usage Factor 10%
Altmetrics 1%
Impact Factor is published by Thomson Reuters, Altmetrics were least well known
Most widely known by researchers

5. Awarenes
s
5
Q2 Which of these do you think are most useful at measuring research quality? (Select up to 3)
64%
29%
29%
28%
58%
37%
34%
42%
0% 20% 40% 60% 80% 100%
Impact factor (n=2,530)
SNIP (n=51)
SJR (n=126)
Eigenfactor (n=285)
h-index (n=1,335)
Journal Usage Factor (n=309)
F1000 (n=155)
Altmetrics (n=41)
* Only people who said they were aware of a particular metric in Q1 were given the opportunity to
select that metric in Q2, *See appendix for background and approach. Research by Elsevier Research
& Academic Relations. Impact Factor is published by Thomson Reuters,
TOTAL
(n=3,090)
82%
2%
4%
9%
43%
10%
5%
1%
Researcher perception of most useful
% useful

6. Generally metrics with the highest awareness are
also considered to be the most useful
6
Impact factor
SNIP
SJR
Eigenfactor
h-index
Journal Usage Factor
F1000
Altmetrics
R² = 0.697
-10%
0%
10%
20%
30%
40%
50%
60%
70%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Percentageofawarerespondentsthatchosethe
metricasoneofthemostuseful
Percentage of respondents that are aware of the metric
The trendline shows the linear
trend for the relationship
between awareness and usage
of metrics
Metrics above the line have
lower levels of awareness, but
are more likely to be rated as
useful than the typical
awareness-usage relationship
Metrics below the line have
higher levels of
awareness, but are less likely
to be rated as useful than the
typical awareness-usage
relationship
*See appendix for background and approach. Research by Elsevier Research & Academic Relations.
Impact Factor is published by Thomson Reuters,

7. 7
Assessing the usefulness of potential quality
metrics: by age
 Significant difference between
subset and total (subset higher)
 Significant difference between
subset and total (subset lower)
Under 36 (n=540) 36-45 (n=920) 46-55 (n=819) 56-65 (n=507) Over 65 (n=242)
TOTAL
(n=3,090)
Article
views/downloads (for
articles)
 43%
Citations from
materials that are in
repositories
  43%
Share in social
network mentions (for
articles)
   16%
Number of readers
(for articles)  40%
Number of followers
(for researchers)   31%
Votes or ratings (for
articles)   24%
A metric that measures
the contribution an
individual makes to peer
review (for researchers)
 28%
A score based on
reviewer assessment (for
articles)
 28%
Q3 Thinking about possible new measures of research productivity, how useful do you think the below would be in assessing the quality of a
researcher or a research article?(By age) % Think it would be extremely/very useful
43%
49%
21%
42%
38%
35%
34%
33%
44%
45%
18%
41%
33%
24%
29%
29%
45%
41%
15%
39%
28%
22%
27%
27%
44%
41%
12%
41%
30%
22%
26%
27%
36%
37%
13%
35%
30%
19%
24%
27%

8. 8
Assessing the usefulness of potential
quality metrics: by region (1 of 2)
 Significant difference between
subset and total (subset higher)
 Significant difference between
subset and total (subset lower)
Africa
(n=72)
APAC
(n=803)
Eastern Europe
(n=183)
Latin America
(n=182)
TOTAL
(n=3,090)
Article views/downloads (for
articles)     43%
Citations from materials that are
in repositories    43%
Share in social network mentions
(for articles)    16%
Number of readers (for articles)  40%
Number of followers (for
researchers)   31%
Votes or ratings (for articles)    24%
A metric that measures the
contribution an individual makes
to peer review (for researchers)
  28%
A score based on reviewer
assessment (for articles)    28%
Q3 Thinking about possible new measures of research productivity, how useful do you think the below would be in assessing the quality
of a researcher or a research article? (By region, slide 1 of 2) % Think it would be extremely/very
useful
56%
51%
26%
49%
36%
33%
40%
44%
50%
55%
27%
46%
46%
29%
35%
36%
50%
49%
19%
45%
41%
30%
28%
26%
50%
49%
21%
45%
34%
24%
32%
35%

9. 9
Assessing the usefulness of potential quality
metrics: by region (2 of 2)
 Significant difference between
subset and total (subset higher)
 Significant difference between
subset and total (subset lower)
Middle East (n=47) North America (n=770) Western Europe (n=1,033)
TOTAL
(n=3,090)
Article views/downloads (for
articles)  43%
Citations from materials that
are in repositories  43%
Share in social network
mentions (for articles)   16%
Number of readers (for articles)   40%
Number of followers (for
researchers)   31%
Votes or ratings (for articles)   24%
A metric that measures the
contribution an individual
makes to peer review (for
researchers)
 28%
A score based on reviewer
assessment (for articles)  28%
Q3 Thinking about possible new measures of research productivity, how useful do you think the below would be in assessing the quality of
a researcher or a research article? (By region, slide 2 of 2) % Think it would be extremely/very
useful
40%
40%
19%
43%
32%
28%
32%
34%
41%
42%
10%
36%
23%
19%
26%
26%
36%
32%
11%
36%
23%
22%
23%
22%