O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Vincent Larivière - Journal Impact Factors: history, limitations and adverse effects

33 visualizações

Publicada em

This panel addresses the application of bibliometric and scientometric methods applied in the assessment of journals and the research they publish.

The measurement of impact of scientific journals through citations has its origin in the documentalists activities at the late nineteenth century to organize the publications of specific areas. The unfolding of these efforts soon undertook quantitative approaches aiming at understanding trends which allowed us to establish, for example, the nucleus of journals and authors in the various areas, making it an important input for science historians and sociologists.

Regarding the treatment of scientific information, the essays of the first half of the twentieth century were materialized into a system that would offer a new form of information retrieval – in the diachronic sense – allowing to identify the relation that literature establishes from the publishing of an article. This relationship, which expresses the repercussion of a new knowledge in the literature, did not take much time to attribute the idea of scientific impact, whose expression occurs through citations. The citation index then revolutionizes the way of accessing literature in the second half of that century, at the same time as it becomes a unique source for impact indicators, which from there would represent the world science in evaluative processes around of the world.

At the turn of the twenty first century, many factors – such as subscription costs, the underrepresentation of the scientific literature of non-English-speaking countries, as well as the different practices of scientific communication among areas of knowledge – have given rise to initiatives aimed at broader sources of information, while at the same time facilitating free access to scientific information. However, besides the access issue, the already established need for impact measurement could not be ignored in order to provide the consolidated processes of evaluation of scientific output with more adequate indicators.

In this sense, it is necessary that the new information sources, taking advantage of the new methodologies proposed by the community specialized in quantitative methods of science evaluation, may contribute with indicators that make the assessment of national (Brazilian) scientific output more adequate to the national scenario. In doing so, it is hoped that the group’s discussions will contribute not only to evidence the best that has been produced locally, but also to allow scientific journals edited nationally, particularly those of the SciELO Network, to have their impact recognized, allowing circulation of global and inclusive scientific knowledge.

Syllabus
Information sources for generating impact indicators; specificity of the culture of scientific communication in the different areas, especially the Human and Social Sciences; the limitations of the Impact Factor and mainstream journal-based indicators; methodologies for…

Publicada em: Ciências
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

Vincent Larivière - Journal Impact Factors: history, limitations and adverse effects

  1. 1. Journal Impact Factors: history, limitations and adverse effects Vincent Larivière Cassidy R. Sugimoto vincent.lariviere@umontreal.ca @lariviev crc.ebsi.umontreal.ca
  2. 2. What will this guy talk about? •Brief history of journal-level indicators •ISI and the Journal Citation Report (JCR) •Critiques of the JCR •Adverse effects •Why do we need journal level indicators?
  3. 3. Gross & Gross (1927) College Libraries and Chemical Education • Chemistry professors (US college) o “What files of scientific periodicals are needed in a college library to successfully prepare the student for advance work, taking into consideration also those materials necessary for the stimulation and intellectual development of the faculty?” • Citations provide an “objective” indicator o “One way to answer this question would be merely to sit down and compile a list of those journals which one considers indispensable. Such a procedure might prove eminently successful in certain cases, but it seems reasonably certain that often the result would be seasoned too much by the needs, likes and dislikes of the compiler.”
  4. 4. Gross & Gross (1927) • Compiled references made by 1926 papers published in the Journal of the American Chemical Society. o Excluded journal self-citations o Excluded cited papers from 1926 (to account for diffusion lag) • This resulted in 3,633 references to 247 journals.
  5. 5. Follow-up analyses • Several papers (≈20) scaled up the method devised by Gross & Gross • Growing datasets (i.e. manual work) • Mostly mono-disciplinary • All from the U.S. Archambault, É., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635-649.
  6. 6. Institute for Scientific Information • First large-scale citation index • Inspired on Shepard’s Citations—created in 1873 for legal research • Science Citation Index (1961) o NSF and NIH funded; cost of 400,000$ o Genetics Citation Index (subset) o Officially launched in 1964 • Sample of most cited journals (Bradford’s Law)
  7. 7. • Garfield and Sher (1963): exploration of the 1961 citation data (1.4 million references) • First mention of a Journal Impact Factor: o “One of the most interesting correlations is the “journal impact factor.” In the usual citation count methods, as, e.g., Gross and Gross (4), the importance of n journal is determined by the absolute number of citations to it.” o “The first step in obtaining a more meaningful measure of importance is to divide the number of times a journal is cited by the number of articles that journal has published.” Institute for Scientific Information (2)
  8. 8. Garfield gets scooped! • Martyn, J., Gilchrist, A. (1968), An Evaluation of British Scientific Journals. Aslib. • Based on the Science Citation Index! o “All citations to 1,842 British journals in the broad areas of science and technology which appeared in the 1965 Science Citation Index for the years 1964 and 1963.” • Data by discipline • Asymmetry between numerator and denominator o All citations / citable items • 2 year citation window
  9. 9. 1975 Journal Citation Report • First Journal Citation Reports (JCR) and reporting of the Journal Impact Factor • Based on more than 4.2 million references made in 1974 by 400,000 papers published in about 2,400 journals • Justifies citations as indicators of usage: o “The more frequently a journal’s articles are cited, the more the world’s scientific community implies that it finds the journal to be a carrier of useful information” • For librarians: decisions about subscriptions • For researchers: identifying publication venues, especially multidisciplinarity ones • Garfield did not advocate for using the JCR to identify elite journals.
  10. 10. Impact Factor • Average number of citations received by articles published in a journal two years after their publication • Impact Factor of a given journal in 2017 would be calculated as follows: Number of citations received in 2017 by documents published in the journal in 2015-2016 ______________________________________ Number of articles and reviews published in the journal in 2016-2017
  11. 11. Asymmetry numerator / denominator Number and proportion of citations received by articles, reviews, non-citable items, and unmatched citations, for four journals from the field of biochemistry and molecular biology, as well as Nature and Science, 2014-2015 papers and 2016 citations Article Review Non-Citable Items Unmatched Citations N % N % N % N % Cell 20,885 78.6% 3,068 11.5% 601 2.3% 2,016 7.6% 869 27.564 30.410 10.3% Nat. Chem. Biol. 3,263 77.4% 378 9.0% 217 5.1% 356 8.4% 268 13.586 15.066 10.9% PLOS Biol. 3,088 85.3% 6 0.2% 237 6.5% 290 8.0% 384 8.057 9.797 21.6% FASEB J. 3,650 74.6% 235 4.8% 203 4.2% 802 16.4% 881 4.410 5.498 24.7% Nature 55,380 78.6% 3,925 5.6% 5,067 7.2% 6,047 8.6% 1,784 33.243 40.140 20.7% Science 45,708 73.0% 4,886 7.8% 5,657 9.0% 6,340 10.1% 1,721 29.398 37.210 26.6% Journal N. Citable Items JCR Impact Factor Symmetric Impact Factor % Increase
  12. 12. Inclusion of journal self-citations Percentage of journal self-citations, by discipline, for citations received in 2016 by papers published in 2014-2015 0% 5% 10% 15% 20% 25% All disciplines Biomedical Research Clinical Medicine Chemistry Mathematics Psychology Biology Health Social Sciences Engineering and Technology Physics Earth and Space Professional Fields Humanities Arts Percentage ofjournal self-citations
  13. 13. Skewness of distributions Distribution of citations received by articles and reviews, for four journals from the field of biochemistry and molecular biology, 2014-2015 papers and 2016 citations 0 5 10 15 20 25 30 35 40 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Cell (JIF = 30.410) 0 5 10 15 20 25 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations Nat. Chem. Biol. (JIF = 15.066) 0 5 10 15 20 25 30 35 40 45 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations PLOS Biol. (JIF = 9.797) 0 20 40 60 80 100 120 140 160 0 10 20 30 40 50 60 70 80 90 100+ Numberofpapers Number of citations FASEB J. (JIF = 5.498)
  14. 14. Skewness of distributions (2) Distribution of the number of journals, by proportion of papers that obtained the JIF value, 2014-2015 papers and 2016 citations 0 100 200 300 400 500 600 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Numberofjournals Proportion ofpapers thatobtained the JIF value
  15. 15. Length of citation window Number of citations (left panel) and cumulative number of citations (right panel), by year following publication for papers published in 1985. 0 0.5 1 1.5 2 2.5 3 3.5 0 5 10 15 20 25 30 Citations Year following publication Number of citations 0 5 10 15 20 25 30 35 40 45 0 5 10 15 20 25 30 Citations Year following publication Cumulative citations 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 5 Citations Yea C 0 5 10 15 20 25 30 35 40 45 0 5 10 15 20 25 30 Citations Year following publication Cumulative citations 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 5 10 15 20 25 30 Citations Year following publication Cumulative % of citations Biomedical Research Psychology Physics Social Sciences
  16. 16. Disciplinary comparison Mean and maximum JIF of journals, by discipline, 2014-2015 papers and 2016 citations Discipline Mean JCR JIF Maximum JCR JIF Mean N. Ref. M Biology 1.683 22.81 48.99 Biomedical Research 3.526 46.6 48.94 Chemistry 2.768 47.93 46.37 Clinical Medicine 2.976 187.04 41.94 Earth and Space 2.173 30.73 53.71 Engineering and Technology 1.989 39.74 36.35 Health 1.647 17.69 39.08 Mathematics 1.017 9.44 26.56 Physics 2.699 37.85 36.57 Professional Fields 1.565 11.12 53.51 Psychology 2.050 19.95 54.56 Social Sciences 1.199 6.66 49.09
  17. 17. Disciplinary comparison (2) Mean JIF values as a function of A) Number of references and B) Age of cited references R² = 0.523 0 1 2 3 4 5 6 0 10 20 30 40 50 60 MeanImpactFactor Number of references JIF and cited references R² = 0.4721 0 1 2 3 4 5 6 0 10 20 30 40 MeanImpactFactor Age of references JIF and age of references A B
  18. 18. Declining predictive power Correlation between the JIF and the citations received by papers, 2 year following publication y = 0.0011x - 1.9418 r2 = 0.4666 y = 0.003x - 5.7759 r2 = 0.579 y = -0.0049x + 9.9863 r2 = 0.4788 0,00 0,05 0,10 0,15 0,20 0,25 0,30 0,35 1900 1920 1940 1960 1980 2000 2020 Annualcoefficientofdetermination(r2)
  19. 19. Declining predictive power Declining share of top 1% most cited papers for elite journals
  20. 20. Impact Factor inflation
  21. 21. Impact Factor inflation
  22. 22. Impact Factor inflation
  23. 23. Impact Factor inflation Impact Factor by journal as a function of rank, for years 1997, 2007, and 2016 0.001 0.01 0.1 1 10 100 1000 1 10 100 1000 10000 100000 ImpactFactor(logscale) Rank of journal (log scale) 2016 2007 1997
  24. 24. Post JIF indicators • Five-year Impact Factor (Clarivate) • Eigenfactor (Clarivate) and SCImago Journal Rank (Scopus) • Source Normalized Impact per Paper—SNIP (Scopus) • CiteScore • None of those indicators solve the main misuse of the JIF: the application of a journal-level indicator to papers and researchers
  25. 25. Systemic effects • Orients researchers’ publication strategies o and research topics o Weakens non-indexed journals • Impact Factor engineering o Optimizing citation window o Numerator/denominator and front matter o Coercitive citations • Cash per publication policies o Science/Nature article  20 times mean academic salary (China) o Disciplinary inequalities… • Multiplication of citation indicators o Anyone can find an indicator where one looks better o Knock off indicators
  26. 26. Knock off Impact Factors
  27. 27. Knock off Impact Factors
  28. 28. Do we need the JIF? What problem does the JIF solve? Information filtering and ranking. Are there other mechanisms for this? What are the effects of the JIF? Journal indicators reify the role of journals—especially anglo-american. Is such hierarchy necessary? The community must ask: Is the use of the Journal Impact Factor good for science? … AND EVEN MORE IMPORTANTLY: Is research evaluation good for science?
  29. 29. Vincent Larivière Cassidy R. Sugimoto vincent.lariviere@umontreal.ca @lariviev crc.ebsi.umontreal.ca Obrigado!
  30. 30. Garfield’s 1972 paper • Citations received during the last quarter of 1969 (Multiplied by four to obtain a year estimate) o First calculation of an Impact Factor (except M & G) • Top 152 journals account for 50% of all references • “It is apparent, even from the makeup of this partial listing, that a good multidisciplinary journal collection need contain no more than a few hundred titles.” • “This analysis gives good reason for concern about any increase in the number of scientific and technical journals.” • Citation Analysis as a Tool in Journal Evaluation. Journals can be ranked by frequency and impact of citations for science policy studies.

×