On the Limits of Bibliometrics

901 visualizações

Publicada em

Apesar de seu uso generalizado, os métodos bibliométricos têm várias limitações, especialmente quando se trata da análise dos países em desenvolvimento. Esta apresentação irá recuperar estas limitações (cobertura parcial da literatura nacional em países de língua não inglesa, baixa cobertura de livros, etc.), bem como fornecer dados sobre a sua extensão nas diferentes disciplinas. Também serão detalhados as limitações do Fator de Impacto e os argumentos contra a sua utilização em avaliação da pesquisa.

Despite their widespread use, bibliometric methods have several limitations, especially when it comes to the analysis of developing countries. This presentation will recall these limitations (partial coverage of national literature for non-English speaking countries, weak coverage of books, etc.) as well as provide data on their extent in the different disciplines. It will also detail the drawbacks of the Impact Factor and argue against its use in research evaluation.

A pesar de su uso generalizado, los métodos bibliométricos tienen varias limitaciones, especialmente cuando se trata del análisis de los países en desarrollo. Esta presentación recordará estas limitaciones (cobertura parcial de la literatura nacional de los países de habla no inglesa, la debilidad de la cobertura de libros, etc), así como proporcionar datos sobre su extensión en las distintas disciplinas. También describirá pormenorizadamente los inconvenientes del Factor de Impacto y argumentar en contra de su uso en la evaluación de la investigación.

Publicada em: Tecnologia, Negócios
0 comentários
1 gostou
Estatísticas
Notas
  • Seja o primeiro a comentar

Sem downloads
Visualizações
Visualizações totais
901
No SlideShare
0
A partir de incorporações
0
Número de incorporações
1
Ações
Compartilhamentos
0
Downloads
7
Comentários
0
Gostaram
1
Incorporações 0
Nenhuma incorporação

Nenhuma nota no slide

On the Limits of Bibliometrics

  1. 1. Faculté des arts et des sciences École de bibliothéconomie et des sciences de l'information On the Limits of Bibliometrics Vincent Larivière Canada Research Chair on the transformations of scholarly communication Université de Montréal SciELO 15 anos 24 Outubro 2013 SciELO 15 anos 1/15
  2. 2. Faculté des arts et des sciences SciELO 15 anos 2/15 Outline of the talk - Technical limits - Coverage - Limits of indicators - Impact Factor and prediction of citations - Citation window - Adverse consequences
  3. 3. Faculté des arts et des sciences SciELO 15 anos Technical limits - No database is covers all of the published literature. - Particularly true in the social sciences and humanities. - Local journals are not indexed by major databases and hence, local topics are not well covered - No books were indexed (until recently). 3/15
  4. 4. Faculté des arts et des sciences SciELO 15 anos Percentage of references made to journal articles in the natural and medical sciences (Larivière et al. 2006) 95% 90% 85% 80% 75% 70% 65% 60% 55% 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 Year Average SCI Physics Chemistry Engineering Biomedical Research Biology Psychology Clinical Medicine Earth and Space Sciences Mathematics 4/15
  5. 5. Faculté des arts et des sciences SciELO 15 anos Percentage of references made to journal articles in the social sciences and humanities (Larivière et al. 2006) 5/15
  6. 6. Faculté des arts et des sciences SciELO 15 anos Proportion of references made to conference proceedings in the natural and medical sciences (Lisée et al. 2008) 6/15 Computers Electrical Engineering & Electronics Civil Engineering Nuclear Technology Miscellaneous Engineering & Technology Aerospace Technology Industrial Engineering Mechanical Engineering Acoustics Human Factors Library & Information Science Chemical Engineering Optics General Engineering Metals & Metallurgy Applied Physics 0% 5% 10% 15% 20%
  7. 7. Faculté des arts et des sciences SciELO 15 anos Coverage of journals as a function of the language of publication (Archambault et al. 2006) NSE Language SSH Thomson ISI Ulrich Difference Thomson ISI Ulrich Difference English 89% 78% 13% 90% 75% 20% Czech 0.04% 0.3% -85% 0.2% 0.2% 8% Russian 0.5% 0.9% -48% 0.3% 0.4% -24% French 3.3% 3.4% -3% 3.2% 4.4% -26% Multiple languages 0.2% 0.2% -14% 0.3% 0.5% -45% Dutch 2.2% 2.2% 0% 1.3% 2.6% -48% German 3.2% 3.9% -18% 3.0% 5.8% -50% Japanese 0.4% 1.7% -74% 0.2% 0.6% -64% Swedish - 0.1% -100% 0.1% 0.4% -69% Spanish 0.6% 2.6% -75% 0.9% 3.0% -69% Italian 0.1% 0.8% -83% 0.2% 1.1% -80% Danish 0.04% 0.1% -50% 0.1% 0.3% -83% Portuguese 0.1% 0.7% -85% 0.1% 1.0% -86% Chinese 0.3% 2.4% -88% 0.04% 1.2% -96% Polish 0.05% 0.7% -92% - 0.9% -100% Arabic - 0.1% -100% - 0.3% -100% Turkish 0.01% 0.2% -95% - 0.1% -100% Other 0.3% 1.6% -80% 0.3% 2.4% -87% 7/15
  8. 8. Faculté des arts et des sciences SciELO 15 anos Median age of cited literature (Archambault et Larivière, 2010) 16 8/15 Arts et humanités 14 12 Sciences sociales 10 8 Sciences naturelles et génie 6 4 c u r t l n a i d é m e g  2 Médecine 0 1980 1985 1990 Année 1995 2000
  9. 9. Faculté des arts et des sciences SciELO 15 anos How to correct (some of) these limits - Creation of book databases - Book Citation Index (but it still has an angloamerican bias!) - Triangulate data sources - Scopus vs. Thomson vs. Google Scholar - Creation of local databases - SciELO in Brazil - Érudit au Québec 9/15
  10. 10. Faculté des arts et des sciences SciELO 15 anos 10/15 Impact factor and its limits  Cannot be compared across disciplines  Skewness of citation distributions  Assymetry between numerator and denominator  Short citation window – especially for SSH  Includes journal self-citations  Is less and less correlated with citations received by papers
  11. 11. Faculté des arts et des sciences SciELO 15 anos 11/15 Different citation practices: mean citation rates of papers after 2 and 10 years, 1900-2005 Mean - 2 year citation window 7 6 5 24 22 MED MED 20 18 NSE 16 4 Mean - 10 year citation window 14 SS 12 3 NSE SS 10 8 2 6 4 1 2 0 1900 1920 1940 1960 1980 2000 0 1900 1920 1940 1960 1980 2000
  12. 12. Faculté des arts et des sciences SciELO 15 anos 12/15 Weakening relationship between Impact Factor and citations (Lozano et al. 2012) 0,35 y = 0.003x - 5.7759 r2 = 0.579 0,30 0,25 0,20 0,15 y = -0.0049x + 9.9863 r2 = 0.4788 ) 2 ( m r d it f e o lc a u n A 0,10 0,05 0,00 1900 y = 0.0011x - 1.9418 r2 = 0.4666 1920 1940 1960 1980 2000 2020
  13. 13. Faculté des arts et des sciences SciELO 15 anos 13/15 Decline of top journals (Larivière et al. 2013)
  14. 14. Faculté des arts et des sciences SciELO 15 anos Adverse effect - Change of research topics Type of journal Brazilian Journals USA Journals Other Journals Brazil in abstract 31% 15% 22% 14/15
  15. 15. Faculté des arts et des sciences SciELO 15 anos 15/15 Bibliography Archambault, É., Vignola-Gagné, É., Côté., Larivière, V. and Y. Gingras (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases, Scientometrics, 68(3): 329-342. Larivière, V., Archambault, É, Gingras, Y. and É. Vignola-Gagné (2006). The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities, Journal of the American Society for Information Science and Technology, 57(8): 9971004. Larivière, V., Archambault, É., Gingras, Y. (2008) Long-term variations in the aging of scientific literature: from exponential growth to steady-state science (1900-2004). Journal of the American Society for Information Science and Technology, 59(2): 288–296. Larivière, V., Gingras, Y. and É. Archambault (2006). Canadian Collaboration Networks: A Comparative Analysis of the Natural Sciences, Social Sciences and the Humanities, Scientometrics, 68(3): 519-533. Larivière, V. , Lozano, G.A., Gingras, Y. (2013). Are elite journals declining? Forthcoming in the Journal of the American Society for Information Science and Technology. Lozano, G.A., Larivière, V., Gingras, Y. (2012) The weakening relationship between the Impact Factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology. 63(11): 2140-2145. Lisée, C., Larivière, V., Archambault, É. (2008) Conference proceedings as a source of scientific information: A bibliometric analysis. Journal of the American Society for Information Science and Technology, 59(11): 1776–1784.

×