Citation analysis for research evaluation

1. Citation analysis for research evaluation Wouter Gerritsma, Marc Loman, Marianne Renkema Information Specialists, Wageningen UR Library

2. Programme <ul><li>Introduction </li></ul><ul><li>Citation data </li></ul><ul><li>Benchmarking </li></ul><ul><li>Journal quality </li></ul><ul><li>Ranking University </li></ul><ul><li>On publishing </li></ul>

3. Why research evaluation/citation analysis? <ul><li>Research assement of graduate schools (5 year cycle, stipulated by the VSNU) </li></ul><ul><ul><li>Other research groups are reviewed in a similar way </li></ul></ul><ul><ul><li>Benchmarking becomes more important </li></ul></ul><ul><li>Grant applications </li></ul><ul><li>Job positions </li></ul><ul><li>In the press </li></ul>Sooner of later, you will be subjected to such an analysis

4. Objective of this course <ul><li>How are citation analyses performed, and what are their shortcomings or weaknesses? </li></ul><ul><li>What can you do to enhance your results </li></ul>

5. Citation data

6. Citation data <ul><li>Paid subscription (wide coverage) </li></ul><ul><ul><li>Web of Science </li></ul></ul><ul><ul><li>Scopus </li></ul></ul><ul><li>Free systems broad coverage </li></ul><ul><ul><li>Google Scholar </li></ul></ul><ul><li>Paid subscription (specialized coverage) </li></ul><ul><ul><li>SciFinder </li></ul></ul><ul><ul><li>Psychindex </li></ul></ul><ul><li>Free systems (specialized coverage) </li></ul><ul><ul><li>Citeseer </li></ul></ul><ul><ul><li>Citebase </li></ul></ul>

7. Web of Science <ul><li>Thomson Scientific </li></ul><ul><ul><li>Founder Eugene Garfield </li></ul></ul><ul><ul><li>Formerly: Institute of Scientific Information (ISI) </li></ul></ul><ul><li>Indexes articles of about 8700 journals, standard bibliographic information + reference lists </li></ul><ul><li>Sell the same data in various databases to libraries and others (CC, WoS, JCR, ESI) </li></ul><ul><li>They (still) have monopoly position for citation data; they are the Golden Standard </li></ul>

8. CC, WoS, JCR, ESI <ul><li>CC: Current Contents, weekly updated bibliographic database covering the top of Sciences (no citation data) </li></ul><ul><li>WoS: Web of Science, weekly updated database, that includes references and citation data as well (slower than CC) </li></ul><ul><li>JCR: Journal Citation Report, yearly analysis of WoS data, presenting perfomance measures for journals. </li></ul><ul><li>ESI, Essential Science Indicators, bimonthly updated analytical database giving performance measures for Countries, Institutes, Scientists and Journals next to research fronts, hot papers and baselines. </li></ul>

9. Database Coverage WoS/JCR/CC Source: Moed (2005) Mathematics / Engineering / Economics Other social sciences related to medicine & health Physics & Astronomy Psychology & psychiatry Clinical medicine Humanities Biological Sciences related to plants & animals Biological Sciences related to humans Other Social Sciences Applied physics & Chemistry Molecular biology & Biochemistry Moderate Good Excellent

10. Scopus <ul><li>(New) product of Elsevier Science </li></ul><ul><li>Covers 15,000 serials </li></ul><ul><ul><li>Not all of them peer reviewed </li></ul></ul><ul><li>Coverage of citation data 1996 -> </li></ul>

11. Google Scholar <ul><li>Free database </li></ul><ul><li>Some concerns about data quality </li></ul><ul><li>Unknown source base </li></ul><ul><li>Very labour intensive to extract data </li></ul>

12. Why Scopus doesn't add substantially to the number of citations found in WoS Source: http://www.wowter.nl/blog/2007/01/why-scopus-doesnt-add-substantially-to.html

13. Searching WoS <ul><li>General search: Searches bibliographic data: Title, Abstract, keywords, Authors, Addresses </li></ul><ul><ul><li>Not all citations are found due to errors in citing, indexing etc. </li></ul></ul><ul><li>Cited reference search: searches in the references of source data </li></ul><ul><ul><li>Not all articles are found. Non cited articles are missing </li></ul></ul>

15. Exercises !

16. Benchmarking

17. Benchmarking <ul><li>Essential Science Indicators (ESI) </li></ul><ul><li>h-index </li></ul>

18. How do we compare numbers? <ul><li>Scientist Z. Math has a publication from 1996 with 17 citations </li></ul><ul><li>Scientist M. Biology has a publication from 2003 with 24 citations </li></ul>

19. Baseline mathematics

20. Baseline Molecular Biology

21. Essential Science Indicators (ESI) <ul><li>Analytical database of SCI, covering 10 years + current year building </li></ul><ul><li>Comparison between Countries, Institutes, Scientists and Journals </li></ul><ul><li>Hot papers / Highly cited papers </li></ul><ul><li>Research fronts </li></ul><ul><li>Baselines </li></ul>

22. Publication age : world averages citations (1)

23. In practice

24. Relative impact of a research institute 1,84 2,26 2,08 2,06 All subfields 1,73 0.57 1,70 Microbiology 1,11 1,81 1,73 Clinical medicine 1,76 1,76 Chemistry 1,09 0,44 1,55 0,91 Biology & biochemistry 3,60 3,87 3,86 3,82 Agricultural Sciences Group 3 Group 2 Group 1 All groups Subfield

25. Evaluation of candidates 0 3 0,83 0,58 0,75 346 57 e 3 16 1,94 1,69 1,86 1886 88 d 0 8 0,9 1,39 1,15 972 93 c 1 17 1,95 1,84 1,93 498 65 b 2 4 1,52 1,76 1,64 1565 80 a #papers top 1% #papers top 10% RI 1999-2003 RI 1994-1998 Relatieve Impact #Citations # Papers 1994-2003 Author

26. h-index <ul><li>A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np − h) paper have no more than h citations each. </li></ul>

27. h-index <ul><li>h-index tries to find a balance between productivity and citation impact </li></ul><ul><li>Only published in 2005, has made a substantial impact in the world of bibliometrics. </li></ul><ul><li>Implemented in Spires, WoS (and soon) in Scopus </li></ul><ul><li>Applicable to authors, journals, research groups, compounds, subjects etc … </li></ul>

28. Exercises !

29. Journal quality

30. Journal quality <ul><li>JCR (Impact factors) </li></ul><ul><li>ESI </li></ul><ul><li>h-index </li></ul><ul><ul><ul><li>Or pagerank ? </li></ul></ul></ul>

31. Warning ! <ul><li>It has often been attempted to use Journal Impact factors in some or other way to judge the performance of a group or individual scientists. </li></ul>This is proven very bad practice!

32. Impact factor <ul><li>Performance measure for journals </li></ul><ul><ul><li>“… it is also used for assessment of the quality of individual papers, scientists and departments. For the latter a scientific basis is lacking, as we will demonstrate in this contribution” (Opthof, 1997) </li></ul></ul><ul><ul><li>Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research 33 (1): 1-7. http:// dx.doi.org /10.1016/S0008-6363(96)00215-5 </li></ul></ul>

33. Journal Citation Reports (JCR) <ul><li>Reports 3 measures </li></ul><ul><li>Impact factor </li></ul><ul><li>Immediacy Index </li></ul><ul><li>Cited half life </li></ul>Cited half-life 50% citations 50% citations 3 1 Immediacy index Window Impact Factor Window

34. Definition of Impact factor <ul><li>The impact factor of a journal J in year T is defined as follows: </li></ul><ul><li>“ The number of citations received in year T by all documents published in J in the years T-1 and T-2” </li></ul><ul><li>÷ </li></ul><ul><li>“ The number of citable documents published in J in the years T-1 and T-2” </li></ul>

35. Citable items <ul><li>Cited items (nominator) include: </li></ul><ul><ul><li>Research articles, review arrrticles, notes, letters, editorials, news items, corrections and meeting abstracts </li></ul></ul><ul><li>Citable items (by definition) </li></ul><ul><ul><li>Articles, notes and reviews </li></ul></ul>

36. Lancet (2002) JCR Like IF = (b)/(a) = 10.2 2.4 15,670(b) 6,443 Total 0.5 2,564 4,899 Other types 8.5 13,106 1,544 (a) Articles/reviews (citable items) Cites/Doc Cites No. Docs Type of document

37. 50 % of articles generate 90% of all cites Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ 314 (7079): 497-502. http://bmj.bmjjournals.com/cgi/content/full/314/7079/497

38. Some other points <ul><li>IF measures citation impact of articles in 2 nd & 3 rd year after publication. Therefore biased towards journals showing rapid maturing. </li></ul><ul><li>Reference practices, particularly the number of references per article and their age distribution, vary considerably among subfields. IF between subfields can not be compared . </li></ul>

39. When do you make use of IF <ul><li>Answering the questions where do you want to publish your manuscript. </li></ul><ul><li>Other questions, to be asked </li></ul><ul><ul><li>Reach colleagues ? </li></ul></ul><ul><ul><li>In which article indexes included? </li></ul></ul><ul><ul><li>Open access? </li></ul></ul>

40. Exercises !

41. Ranking universities <ul><li>Popular lists </li></ul><ul><ul><li>Shanghai Jiao Tong University </li></ul></ul><ul><ul><li>THES </li></ul></ul><ul><ul><li>Newsweek </li></ul></ul><ul><li>ISIhighly cited </li></ul><ul><li>ESI </li></ul><ul><li>Other reports </li></ul><ul><ul><li>EC, NOWT, OECD, USA </li></ul></ul>

42. Winding down

49. What’s in a name? <ul><li>A.G.J. Voragen or A. Voragen or F. Voragen or F.A.G.J. Voragen or F.G.J. Voragen </li></ul>