Journal Impact Factors:
history, limitations and
adverse effects
Vincent Larivière
Cassidy R. Sugimoto
vincent.lariviere@umontreal.ca
@lariviev
crc.ebsi.umontreal.ca

What will this guy talk about?
•Brief history of journal-level indicators
•ISI and the Journal Citation Report (JCR)
•Critiques of the JCR
•Adverse effects
•Why do we need journal level indicators?

Gross & Gross (1927)
College Libraries and Chemical Education
• Chemistry professors (US college)
o “What files of scientific periodicals are needed in a college library to
successfully prepare the student for advance work, taking into
consideration also those materials necessary for the stimulation and
intellectual development of the faculty?”
• Citations provide an “objective” indicator
o “One way to answer this question would be merely to sit down and
compile a list of those journals which one considers indispensable.
Such a procedure might prove eminently successful in certain cases,
but it seems reasonably certain that often the result would be
seasoned too much by the needs, likes and dislikes of the compiler.”

Gross & Gross (1927)
• Compiled references made by 1926 papers published in
the Journal of the American Chemical Society.
o Excluded journal self-citations
o Excluded cited papers from 1926 (to account for diffusion lag)
• This resulted in 3,633 references to 247 journals.

Follow-up analyses
• Several papers (≈20) scaled up the method devised by
Gross & Gross
• Growing datasets (i.e. manual work)
• Mostly mono-disciplinary
• All from the U.S.
Archambault, É., & Larivière, V. (2009). History of the journal impact factor: Contingencies and
consequences. Scientometrics, 79(3), 635-649.

Institute for Scientific Information
• First large-scale citation index
• Inspired on Shepard’s Citations—created in
1873 for legal research
• Science Citation Index (1961)
o NSF and NIH funded; cost of 400,000$
o Genetics Citation Index (subset)
o Officially launched in 1964
• Sample of most cited journals (Bradford’s
Law)

• Garfield and Sher (1963): exploration of the
1961 citation data (1.4 million references)
• First mention of a Journal Impact Factor:
o “One of the most interesting correlations is the
“journal impact factor.” In the usual citation count
methods, as, e.g., Gross and Gross (4), the
importance of n journal is determined by the absolute
number of citations to it.”
o “The first step in obtaining a more meaningful measure
of importance is to divide the number of times a journal
is cited by the number of articles that journal has
published.”
Institute for Scientific Information (2)

Garfield gets scooped!
• Martyn, J., Gilchrist, A. (1968), An Evaluation of
British Scientific Journals. Aslib.
• Based on the Science Citation Index!
o “All citations to 1,842 British journals in the broad areas of science and
technology which appeared in the 1965 Science Citation Index for the
years 1964 and 1963.”
• Data by discipline
• Asymmetry between numerator and denominator
o All citations / citable items
• 2 year citation window

1975 Journal Citation Report
• First Journal Citation Reports (JCR) and reporting of the
Journal Impact Factor
• Based on more than 4.2 million references made in 1974
by 400,000 papers published in about 2,400 journals
• Justifies citations as indicators of usage:
o “The more frequently a journal’s articles are cited, the more the world’s
scientific community implies that it finds the journal to be a carrier of
useful information”
• For librarians: decisions about subscriptions
• For researchers: identifying publication venues, especially
multidisciplinarity ones
• Garfield did not advocate for using the JCR to identify
elite journals.

Impact Factor
• Average number of citations received by articles
published in a journal two years after their
publication
• Impact Factor of a given journal in 2017 would be
calculated as follows:
Number of citations received in 2017 by documents published
in the journal in 2015-2016
______________________________________
Number of articles and reviews
published in the journal in 2016-2017

Asymmetry numerator / denominator
Number and proportion of citations received by articles,
reviews, non-citable items, and unmatched citations, for four
journals from the field of biochemistry and molecular biology, as
well as Nature and Science, 2014-2015 papers and 2016
citations
Article Review
Non-Citable
Items
Unmatched
Citations
N % N % N % N %
Cell 20,885 78.6% 3,068 11.5% 601 2.3% 2,016 7.6% 869 27.564 30.410 10.3%
Nat. Chem. Biol. 3,263 77.4% 378 9.0% 217 5.1% 356 8.4% 268 13.586 15.066 10.9%
PLOS Biol. 3,088 85.3% 6 0.2% 237 6.5% 290 8.0% 384 8.057 9.797 21.6%
FASEB J. 3,650 74.6% 235 4.8% 203 4.2% 802 16.4% 881 4.410 5.498 24.7%
Nature 55,380 78.6% 3,925 5.6% 5,067 7.2% 6,047 8.6% 1,784 33.243 40.140 20.7%
Science 45,708 73.0% 4,886 7.8% 5,657 9.0% 6,340 10.1% 1,721 29.398 37.210 26.6%
Journal
N.
Citable
Items
JCR
Impact
Factor
Symmetric
Impact
Factor
%
Increase

Inclusion of journal self-citations
Percentage of journal self-citations, by discipline, for
citations received in 2016 by papers published in
2014-2015
0% 5% 10% 15% 20% 25%
All disciplines
Biomedical Research
Clinical Medicine
Chemistry
Mathematics
Psychology
Biology
Health
Social Sciences
Engineering and Technology
Physics
Earth and Space
Professional Fields
Humanities
Arts
Percentage ofjournal self-citations

Skewness of distributions
Distribution of
citations received
by articles and
reviews, for four
journals from the
field of
biochemistry and
molecular biology,
2014-2015
papers and 2016
citations
0
5
10
15
20
25
30
35
40
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Cell (JIF = 30.410)
0
5
10
15
20
25
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
Nat. Chem. Biol. (JIF = 15.066)
0
5
10
15
20
25
30
35
40
45
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
PLOS Biol. (JIF = 9.797)
0
20
40
60
80
100
120
140
160
0 10 20 30 40 50 60 70 80 90 100+
Numberofpapers
Number of citations
FASEB J. (JIF = 5.498)

Skewness of distributions (2)
Distribution of the number of journals, by proportion of papers that obtained
the JIF value, 2014-2015 papers and 2016 citations
0
100
200
300
400
500
600
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Numberofjournals
Proportion ofpapers thatobtained the JIF value

Length of citation window
Number of citations (left panel) and cumulative number of
citations (right panel), by year following publication for papers
published in 1985.
0
0.5
1
1.5
2
2.5
3
3.5
0 5 10 15 20 25 30
Citations
Year following publication
Number of citations
0
5
10
15
20
25
30
35
40
45
0 5 10 15 20 25 30
Citations
Year following publication
Cumulative citations
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 5
Citations
Yea
C
0
5
10
15
20
25
30
35
40
45
0 5 10 15 20 25 30
Citations
Year following publication
Cumulative citations
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 5 10 15 20 25 30
Citations
Year following publication
Cumulative % of citations
Biomedical
Research
Psychology
Physics
Social
Sciences

Disciplinary comparison
Mean and maximum JIF of journals, by discipline, 2014-2015 papers and
2016 citations
Discipline
Mean
JCR JIF
Maximum
JCR JIF
Mean
N. Ref.
M
Biology 1.683 22.81 48.99
Biomedical Research 3.526 46.6 48.94
Chemistry 2.768 47.93 46.37
Clinical Medicine 2.976 187.04 41.94
Earth and Space 2.173 30.73 53.71
Engineering and Technology 1.989 39.74 36.35
Health 1.647 17.69 39.08
Mathematics 1.017 9.44 26.56
Physics 2.699 37.85 36.57
Professional Fields 1.565 11.12 53.51
Psychology 2.050 19.95 54.56
Social Sciences 1.199 6.66 49.09

Disciplinary comparison (2)
Mean JIF values as a function of A) Number of references and B) Age of
cited references
R² = 0.523
0
1
2
3
4
5
6
0 10 20 30 40 50 60
MeanImpactFactor
Number of references
JIF and
cited references
R² = 0.4721
0
1
2
3
4
5
6
0 10 20 30 40
MeanImpactFactor
Age of references
JIF and
age of references
A B

Declining predictive power
Correlation between the JIF and the citations received by papers, 2 year
following publication
y = 0.0011x - 1.9418
r2 = 0.4666
y = 0.003x - 5.7759
r2 = 0.579
y = -0.0049x + 9.9863
r2 = 0.4788
0,00
0,05
0,10
0,15
0,20
0,25
0,30
0,35
1900 1920 1940 1960 1980 2000 2020
Annualcoefficientofdetermination(r2)

Declining predictive power
Declining share of top 1% most cited papers for elite journals

Impact Factor inflation

Impact Factor inflation

Impact Factor inflation

Impact Factor inflation
Impact Factor by journal as a function of rank, for years 1997, 2007, and
2016
0.001
0.01
0.1
1
10
100
1000
1 10 100 1000 10000 100000
ImpactFactor(logscale)
Rank of journal (log scale)
2016
2007
1997

Post JIF indicators
• Five-year Impact Factor (Clarivate)
• Eigenfactor (Clarivate) and SCImago Journal
Rank (Scopus)
• Source Normalized Impact per Paper—SNIP
(Scopus)
• CiteScore
• None of those indicators solve the main misuse
of the JIF: the application of a journal-level
indicator to papers and researchers

Systemic effects
• Orients researchers’ publication strategies
o and research topics
o Weakens non-indexed journals
• Impact Factor engineering
o Optimizing citation window
o Numerator/denominator and front matter
o Coercitive citations
• Cash per publication policies
o Science/Nature article  20 times mean academic salary (China)
o Disciplinary inequalities…
• Multiplication of citation indicators
o Anyone can find an indicator where one looks better
o Knock off indicators

Knock off Impact Factors

Knock off Impact Factors

Do we need the JIF?
What problem does the JIF solve?
Information filtering and ranking. Are there other mechanisms for this?
What are the effects of the JIF?
Journal indicators reify the role of journals—especially anglo-american.
Is such hierarchy necessary?
The community must ask:
Is the use of the Journal Impact Factor good for science?
… AND EVEN MORE IMPORTANTLY:
Is research evaluation good for science?

Vincent Larivière
Cassidy R. Sugimoto
vincent.lariviere@umontreal.ca
@lariviev
crc.ebsi.umontreal.ca
Obrigado!

Garfield’s 1972 paper
• Citations received during the last
quarter of 1969 (Multiplied by four
to obtain a year estimate)
o First calculation of an Impact Factor (except
M & G)
• Top 152 journals account for 50%
of all references
• “It is apparent, even from the
makeup of this partial listing, that
a good multidisciplinary journal
collection need contain no more
than a few hundred titles.”
• “This analysis gives good reason
for concern about any increase in
the number of scientific and
technical journals.”
• Citation Analysis as a Tool in Journal Evaluation. Journals can be ranked by
frequency and impact of citations for science policy studies.