This document discusses new tools and indicators for measuring university performance in the current globalized, collaborative, and interdisciplinary science landscape. It summarizes several indicators that can provide insight into trends, impact, specialization, networks, and knowledge diffusion including growth indices, average relative citation, average relative impact factor, and specialization indices. Examples are given showing rankings and analyses of performance at institutional, departmental, national and international levels across various fields.

1. Turning Data into Insight
New Tools for Measuring the University
In the Global Science Network

2. What Constitutes a “U.S. Research University”?
Science is becoming more:
International
Global
Growth Indicator
Collaborative
(15 years)
1 = world average
3.8 to 14.4 (11)
1.2 to 3.8 (47)
Networked
0.7 to 1.2 (92)
0.3 to 0.7 (27
0 to 0.3 (7)
too few papers (67)
Interdisciplinary
International collaborations in science have more than doubled since 1990…
2

3. Counting Within “Boxes” is Limited
 Shift in boundaries (nations, disciplines of science) make science
difficult to measure
 Rankings are widely used but simplistic
 Inputs (R&D spending) are only loosely correlated to outputs
 Surveys are expensive and difficult to reproduce
 Knowledge creation, absorption, diffusion, retention are dynamic and
ever-changing
 Indicators are needed that reflect the movement of knowledge, people,
and information
 …The data are the same but we have found new theoretically-based
uses for them…
3

4. Our Value Add – Quality and Insight
We measure performance - trends, impact, and specialization
 comparative analyses of performance and capability
 international, national, faculty and departmental level positional
analyses
 impact and uptake analyses
 collaborative trends analyses
 strengths, weaknesses, and opportunities analyses
Our analyses are based on the most up-to-date information
available from world-class databases.
4

5. We Measure Performance
Trends
 Growth Index (GI): a measure of the increase in output for an area relative to the
database overall.
Impact
 Average Relative Citation (ARC): an indicator of observed impact of a given entity
calculated from citation counts.
 Average Relative Impact Factor (ARIF): an indicator of expected impact based
on the journals in which papers are published.
Specialization
 Specialization Index (SI): a measure of the research intensity of an entity in a given
area relative to a reference entity in the same area.
5

6. Performance Rankings
Top 25 Institutions in California (by # papers) in the Field of Optics
Institution Papers ARC
 rankings can be California Institute of Technology
NASA
1,809
1,422
1.8
1.2
done based on a Stanford University
Lawrence Livermore Natl. Laboratory
1,098
950
2.3
1.9
variety of indices: UC Berkeley
UC San Diego
885
772
1.6
1.4
University of Southern California 763 1.3
- output Los Alamos National Laboratory
UC Irvine
674
638
1.4
1.9
- ARC/ARIF UC Los Angeles
UC Santa Barbara
634
522
1.5
2.0
- specialization Lawrence Berkeley Natl. Laboratory
UC Davis
376
368
1.8
1.9
US Navy 288 0.9
- growth indices UC Santa Cruz
Intel Corporation
236
209
2.0
2.9
Sandia National Laboratories 176 2.6
Raytheon Corporation 168 0.8
 multi-criteria rankings Lockheed Martin Corporation
IBM
167
167
0.6
2.7
Aerospace Corporation 156 0.7
can also be UC San Francisco
KLA-Tencor
155
151
0.9
0.9
assessed Hewlett-Packard
San Diego State University
150
118
0.9
1.8
6

7. Trends - Output Relative to Size
 For the examples below, the overall number of papers per inhabitant are
relatively similar
 The number of papers produced per billion $ is much lower in California
& Washington
GERD
Population (billion$ & Papers/ Papers/
Entity Papers ARIF ARC
(2003 Estimates) PPP) 103 inhab. Billion $1
(2002)
California 35,484,453 63.2 206,480 5.8 653 1.32 1.61
Washington 6,131,445 12.9 39,290 6.4 608 1.27 1.60
Canada 31,676,100 23.5 161,715 5.1 1,374 1.08 1.17
World 6,297,409,158 1,168.5 3,483,166 0.6 596 1 1
 In return for the higher ‘per paper cost’, a higher impact is observed
(measured by ARC and ARIF)
1Papers from 2001-2005 divided by 2002 GERD in Canadian dollars at purchasing power parities
7

8. Strength of leading Canadian institutions in forestry,
1991–2006 (only the top 20 institutions in absolute output presented)
Institution Papers Publication Trend Impact (ARC) Impact Trend
NRCan-CFS 2,321 +N+N+N+N+ +NN+N+N+N+
UBC 1,562 +N+N+N+N+ +NN+N+N+N+
University of Alberta 1,038 +N+N+N+N+ +NN+N+N+N+
Université Laval 773 +N+N+N+N+ +NN+N+N+N+
University of Toronto 552 +N+N+N+N+ +NN+N+N+N+
BC Ministry of Forests 529 +N+N+N+N+ +NN+N+N+N+
Environment Canada 415 +N+N+N+N+ +NN+N+N+N+
University of Victoria 381 +N+N+N+N+ +NN+N+N+N+
Simon Fraser University 374 +N+N+N+N+ +NN+N+N+N+
University of Guelph 353 +N+N+N+N+ +NN+N+N+N+
University of New Brunswick 344 +N+N+N+N+ +NN+N+N+N+
AAFC 327 +N+N+N+N+ +NN+N+N+N+
McGill University 320 +N+N+N+N+ +NN+N+N+N+
University of Calgary 309 +N+N+N+N+ +NN+N+N+N+
Ontario Ministry of Nat Res 296 +N+N+N+N+ +NN+N+N+N+
Ctr for Forest Res (CFR/CEF) 243 +N+N+N+N+ +NN+N+N+N+
University of Saskatchewan 239 +N+N+N+N+ +NN+N+N+N+
Lakehead University 201 +N+N+N+N+ +NN+N+N+N+
Université de Montréal 198 +N+N+N+N+ +NN+N+N+N+
Minist des Res Nat et Faune (QC) 170 +N+N+N+N+ +NN+N+N+N+
8

9. Network of Co-authorships in Cosmology/Gravity Studies
(Sized by Impact, 2000-2005)
9

10. Indicators can be Created from Networks
The Global Network of Science over 15 Years, based on
all scientific publications in the Web of Science
Network index 1990 network 2000 network 2005 network
Number of nodes 172 192 194
Number of links 1 926 3 537 9400
Size of core component 37 54 66
Network density 0.1310 0.1929 0.2511
Average degree 22.442 36.896 48.649
Average distance 1.954 1.851 1.76
Diameter 3 3 3
Graph betweenness 0.2589 0.1617 0.144
Average clustering coefficient 0.784 0.787 0.789
10

11. Impact & Specialization - International Level
 shows the size of an
entity’s research output
in a given area
 shows the level of
specialization of an entity
in a given area
 shows the impact of an
entity in a given area
 shows an entity’s relative
position vis-à-vis other
countries
11

12. Impact & Specialization – A
Research Institution in a National Context
12

13. Impact & Specialization – Institutional Level
(for Management and Business Finance)
 assess the
performance of
specific faculties,
departments or
subfields
 identify areas of
institutional strength
and weakness
 evaluate
opportunities for
strategic
development
13

14. Mapping Knowledge Networks
(citations within the field of quantum materials)
 citation networks map the
links between researchers
 networks can show
various types of clusters
- subject matter
- geographical
 networks help
characterize collaboration
patterns
14

15. Let Us Sharpen Your Profile
Science-Metrix Corporation
Caroline S. Wagner, Ph.D.
404 Russell Road
Alexandria VA 22301
703-836-4569
Caroline.wagner@science-Metrix.com
15