Using Metrics Beyond
Citations to Demonstrate
the Impact of SciELO
24 September 2018
Stephanie Faulkner | Product Manager
S.Faulkner@Elsevier.com | +1.732.216.5104
Supporting and developing metrics for the full
lifecycle of research
We strive for broader use of
open metrics that can be
used by everyone
Research Metrics at Elsevier
• Altmetrics = Alternative ways of measuring the
use and impact of scholarship
• Altmetrics combine traditional impact measures
(citation counts) with non-traditional measures
• Altmetrics = ALL METRICS
“Altmetrics are measures of
scholarly impact mined from
activity in online tools and
environments” – Jason Priem,
Using Altmetrics to Illustrate the Impact of Open Access
on Graduate Student Research (Barnett, Collister, Chan 2014)
(clicks, downloads, views,
library holdings, video plays)
(bookmarks, code forks, favorites,
(blog/news posts, comments,
reviews, Wikipedia links)
(tweets, likes, shares,
(citation indexes, patent
citations, clinical citations)
Library holdings, Clicks,
Downloads, Views, Video
• Is anyone reading our work?
• Which libraries have our book in their
• Usage is the #1 stat researchers want to know
after Citation counts
• PlumX is the only product that includes Usage
• Captures indicate that someone wants to come
back to the work
• How many time has a book citation been
• The number of times a citation/abstract has
been saved, emailed or printed
• Early indicator of future citations
Blog posts, News posts,
Book reviews, Wikipedia
• This category measures people truly engaging
with your research
• Automatically uncover the conversations about
• Discover feedback, opinions, etc.
Shares, likes, & comments,
• Social media measures how well a publisher or
researcher is promoting their work
• Track the buzz and attention around your
• This is especially important for early career
researchers to measure and understand
• Citations still a standard of long-term impact
• Including citations enables side-by-side
analysis with other metrics categories
Visualizing Impact: Plum Print
• Includes the 5 categories
• Circles dynamically
change size based on
metrics in each category
• Blog posts
• Book chapters
• Clinical Trials
• Conference Papers
• Data Sets
• Source Code
• Theses / Dissertations
• Web Pages
ALL Types of Research Artifacts
Scholarly Data Exhaust – Data Source
Tracking anything is a challenge. To track a piece of research you need a way to identify
it. The Digital Object Identifier (DOI) has been widely adopted to identify articles. But
what about non-article output? Or articles that do not have a DOI? Plum Analytics tracks
as many identifiers as possible, including ISBNs for books, SlideShare IDs for
presentations, YouTube IDs for videos and so much more.
>20 URLS &
libraries hold this
Books are often the seminal work of researchers in Humanities and Social
Sciences. Yet, citations do not do them justice – they never have and
never will. This puts researchers from these disciplines at a disadvantage.
Luckily, there are other ways to measure the impact of books.
What are the
Amazon about this
How many people
added this book to
clicks did this
How many books & book chapters? 15,121,312
Measuring the impact of books is
• In a 2016 study, several bibliometricians analyzed over 70,000 books,
collecting and comparing traditional and non-traditional (PlumX) metrics.
• Metrics varied across types of books and content
• They concluded there isn’t one measure that captures the impact of books
across all types -- you need to look a multiple or different measures
• Social Media Mentions
• As well as Citations
• They also concluded that it is important for publishers to utilize social media
to raise awareness to their books (including text books).
Halevi, G., Nicolas, B., & Bar-Ilan, J. (2016). The Complexity of Measuring the Impact of Books. Publishing Research Quarterly,
32(3), 187-200. doi:10.1007/s12109-016-9464-5
Most frequently mentioned in social media
Most frequently viewed & downloaded
Most frequently mentioned in social media
Most frequently mentioned in reviews
PlumX links directly
• See reviews and
• See individuals who
have read or want to
read the book
Uncover the conversations
the author was not even
Finding insights through the PlumX
Organized by Publisher
Filter and analyze
Quickly find which book has the most
metrics in a specific category
Each metric column
can be sorted
(publication year and
PlumX artifact page
Analytics – Metrics by publication year
Analytics – Metrics by publication year
Working from the
middle out, find the
largest section in the
third ring to see the
book with the most
• Tracking over 67 research artifact types such as articles,
presentations, patents, books, book chapters, datasets, videos,
musical scores, thesis and dissertations and more, from over a 50
• Comprehensive metric data grouped into five key categories, vetted
by our partners and librarians including – Usage, Captures, Mentions,
Social Media and Citations. We are the only provider including article
level usage statistics. Sources of book specific metrics are extensive
and coverage growing.
• Measured versions? Artifact (journal, book, book chapter, datasets,
etc.) level of measurement includes not only the Publisher (Gold), but,
also Green Open Access, Preprint, Aggregated versions and
Abstracting & Index versions.
• We currently have metrics for over 70+ million artifacts in our PlumX
index and these numbers continue to grow—-covering a wide breadth
of the world’s research output.
Find out more information at:
Our Metrics Manifesto
Need to use different metrics and common sense
Decisions should be based on both quantitative and qualitative input
Should always use at least two metrics (more than one way to ‘excellence’)
The methodologies should be open, transparent, valid and replicable
Modern research metrics give a more comprehensive and holistic view of impact. These new metrics are more timely than citation metrics and can keep pace with new formats faster than the entrenched, legacy practices. They are now available to provide fresh insights into how research is interacted with, shared, commented on, promoted, and more. So we define altmetrics as ALL METRICS. And PlumX is the only metrics provider that includes usage.
We developed 5 categories of metrics of how to represent the data exhaust that we are collecting. Usage, Captures, mentions, Social Media and Citations. I will give you a brief overview of each category.
Usage is very important – knowing how many times a particular article or artifact is downloaded, viewed or played (if it’s a video). One of the usage metrics we have for books is how many libraries hold a book.
Captures goes beyond viewing the artifact. Someone is saving it or bookmarking it for later or adding it to Mendeley of CiteULike. This indicates that the person wants to come back to this piece of research. Several studies have shown that captures are really good indicator of future citations. People save and store work that they will come back to when they are ready to write their paper or create their research output.
Mentions show engagement – someone is commenting, blogging or reviewing research or even referencing it in a Wikipedia article. It’s where we say the stories are hidden.
Many people think of altmetrics synonymously with social media. We think social media is a great way of measuring how work is promoted – by the researcher or the publisher.
Citations are a part of the overall impact assessment and are important to consider alongside the other metric types we capture. In addition to traditional citation indexes, PlumX metrics also include clinical, patent and policy citations which are a nice measure of impact in society.
The Plum Print changes dynamically based on each artifact’s metrics. It is like a fingerprint – unique to the specific research output. You can see the color-coding and easily identify which of the 5 categories of metrics has the most or least impact. We’re working towards getting the Plum Print on the Scielo Livros platform.
We track 67 different types of output in our database of over 70 million artifacts and 9.5 billion interactions.
Currently we provide metrics from over 50 different types of sources, and these are just a few examples of what we are measuring. Some metric sources update in near-real time, others daily or weekly.
How do we track the different types of output being generated? We can use an array of unique identifiers from ISBN, to ORCID ID, Slideshare Slideshow ID, of course a DOI. Even a Repository handle URI can be used.
Our technology starts with that single identifier and goes and finds additional identifiers related to the research. It could be an ISBN, publisher URL or DOI. In this example, we started with a single repository URL. From that one URL, we found multiple URLs and other identifiers. In the end we are tracking metrics on 3 identifiers and over 20 URLs. We take all of those identifiers and look for them in our over 50 metrics sources.
We know that measuring the impact of books is challenging as books and book chapters are underserved by traditional citation analysis. PlumX includes inventive new measures for books, including how many libraries hold a book, ebook downloads, reviews from Amazon and GoodReads, Wikipedia references, and more.
In 2016 a study was conducted by several bibliometricians who analyzed metrics on over 70,000 books.
They compared traditional and non-traditional metrics (using PlumX metrics) and grouped books into subject areas. The authors acknowledge other research on books that states in general, books are not cited as often as articles and in in many disciplines books become obsolete much faster than articles.
Abrizah, A., & Thelwall, M. (2014). Can the impact of non-Western academic books be measured? An investigation of Google Books and Google Scholar for Malaysia. Journal of the Association for Information Science and Technology, 65(12), 2498-2508. doi:10.1002/asi.23145
In the study, it was interesting to see which subject areas performed better in specific metric categories that other subject areas. This image illustrates which book subject areas tend to receive the most social media attention. Notice the subject areas with much less social media impact, like Education, Engineering, Ecology and Religion.
The analysis also revealed which book subject areas had the most views and downloads (usage). Notice how the subject areas with lesser social media impact, have better usage. I would imagine with a stronger social media presence, these same areas would have even higher usage.
Finally, a look at subject areas with the most book reviews. Medicine and Business, which had high social media impact, are not even listed here.
Let’s take a look at some book examples on the SciELO platform, starting with this publication with a good number of reviews. From any PlumX artifact page (an artifact is what we call any piece of research output), we not only display the number of reviews,
we link to the reviews and ratings on the platform (Goodreads or Amazon). This allows you to see exactly what people are writing.
In this example, you can see this book has 9 Wikipedia references. PlumX currently covers 4 Wikipedia editions – English, Spanish, Portuguese and Swedish. Each language has unique content – the references are not just translations. We link to the full Wikipedia articles where you can see the reference to the artifact. More languages are coming soon. It’s interesting to see the different types of Wikipedia articles that references this single book.
When I look at usage for this particular publisher, this book has high usage compared to other books on the SciELO platform. You can see that for all of our metric categories, we link back to as many sources as possible. Some of our sources (like EBSCO for example), only provide us with a count. This book has high abstract views on the EBSCOHost platform.
We have loaded all of the SciELO book identifiers into a PlumX Dashboard which you will have access to. A PlumX Dashboard allows you view metrics in a variety of ways and uncover insights about the books. The SciELO books dashboard is organized by publisher. Click on the publisher name to see the list of books and associated metrics. Throughout the dashboard you always have the option to export the metrics and perform your own analyses or use our built in analytic reports. A PlumX Dashboard gives you lots of ways to drill into or find information with several ways to filter – by specific metric type, by artifact type or by publication year range. Let’s take a closer look…
Each column can quickly be sorted to see which books have the highest impact . Clicking on the “plus” symbol below the metric category displays the metrics in more detail. After you sort the column, click on the title of any book to view all of the metrics.
View the source for each metric by clicking on the “down arrow” to expand. We display news and blog mentions, original tweets, Wikipedia references and other details if we have them.
When you click on the Analytics button from the PlumX Dashboard, you will find several prepared reports. The first report - artifacts by publication year -- shows you the last 10 years of metrics in each of our 5 categories so you can identify trends over time. The blue line represents the number of books in the year. The colored bar graphs represent each of our metric categories.
Usage is in a separate graph from citations, captures, social media and mentions because usage tends to be a much larger number compared to the other metric categories.
Mouse over a particular year to see the detailed metrics category counts. Look at the high social media metrics in 2009. Click on any bar graph to go to the filter view and sort by social media to see why this category is so high in this year.
You can see that same book from the previous example published in 2009 by Editora FIOCRUZ had a huge amount of social media compared to all the other books by this publisher in the same year.
Expand the social media column, or click on the title to see the social media metric breakdown.
Another report in the analytics area is the Sunburst. The sunburst provides a nice visualization to find the book with the most overall impact by a particular publisher. Work from the inside ring out to the third ring and click on the largest section to see the book details. You can click on the title in the pop-up window to go to the artifact page.
And don’t forget that you can always export all of the detailed metrics as a spreadsheet by clicking the Export Data button on the dashboard. This will allow you to perform your own analyses, selecting the metrics that you choose to highlight.
To summarize: we track 67 types of research output from over 50 sources, grouping them into five main metric categories. We track not only published versions of research, but preprint, A&I and open access and currently have metrics for over 70 million artifacts.
We look forward to your feedback and questions