presentation at ALA Annual 2016 ALCTS/LITA Electronic Resources Management Interest Group panel “Making it count: Usage statistics and electronic resources management.”
1. “Making it count: Usage statistics and electronic resources management”
COUNTER and SUSHI
Oliver Pesch
Chief Product Strategist
EBSCO Information Services
ALCTS/LITA Electronic Resources Management Interest Group
4. COUNTER...
• A Code of Practice seeking “Consistent, Credible & Comparable”
usage reporting usage for scholarly online information
specifying...
– What reports to provide
– How they should be formatted
– What metrics to included
– Guidance for processing transaction logs
– And, how they should be delivered
• Audit required for compliance
5. “Standard” COUNTER Reports
Report Name Report Description
Book Report 1 Number of Successful Title Requests by Month and Title
Book Report 2 Number of Successful Section Requests by Month and Title
Book Report 3 Access Denied to Content Items by Month, Title and Category
Book Report 4 Access Denied to Content items by Month, Platform and Category
Book Report 5 Total Searches by Month and Title
Database Report 1 Total Searches, Result Clicks and Record Views by Month and Database
Database Report 2 Access Denied by Month, Database and Category
Journal Report 1 Number of Successful Full-Text Article Requests by Month and Journal
Journal Report 1 GOA Number of Successful Gold Open Access Full-Text Article Requests by Month and Journal
Journal Report 2 Access Denied to Full-Text Articles by Month, Journal and Category
Journal Report 5 Number of Successful Full-Text Article Requests by Year-of-Publication (YOP) and Journal
Multimedia Report 1 Number of Successful Full Multimedia Content Unit Requests by Month and Collection
Platform Report 1 Total Searches, Result Clicks and Record Views by Month and Platform
6. “Standard” COUNTER Reports
Report Name Report Description
Book Report 1 Number of Successful Title Requests by Month and Title
Book Report 2 Number of Successful Section Requests by Month and Title
Book Report 3 Access Denied to Content Items by Month, Title and Category
Book Report 4 Access Denied to Content items by Month, Platform and Category
Book Report 5 Total Searches by Month and Title
Database Report 1 Total Searches, Result Clicks and Record Views by Month and Database
Database Report 2 Access Denied by Month, Database and Category
Journal Report 1 Number of Successful Full-Text Article Requests by Month and Journal
Journal Report 1 GOA Number of Successful Gold Open Access Full-Text Article Requests by Month and Journal
Journal Report 2 Access Denied to Full-Text Articles by Month, Journal and Category
Journal Report 5 Number of Successful Full-Text Article Requests by Year-of-Publication (YOP) and Journal
Multimedia Report 1 Number of Successful Full Multimedia Content Unit Requests by Month and Collection
Platform Report 1 Total Searches, Result Clicks and Record Views by Month and Platform
Journal Report 1
Database Report 1
Platform Report 1
Book Reports 1 & 2
7. SUSHI...
• A NISO/ANSI standard (Z39.93) enabling the automated
exchange of COUNTER statistics via web services.
• Client/server protocol
– Client (ERM/usage consolidation module) requests the report via
SOAP web service
– Server responds with COUNTER usage in XML format
– Client processes the usage
• Once configured, usage harvesting can be automatic
8. Brief history of COUNTER and SUSHI
2016 Apr Jun Aug Oct Dec 2017
COUNTER
initiative
launched
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2014 2015 2016
Release 1
Journals &
Databases
Release 2
Journals &
Databases
Release 1
Books
Release 3
Journals &
Databases
Release 4
Journals,
Databases
and Books
SUSHI
initiative
launched
SUSHI Draft
Standard
Released
SUSHI
Standard
Released as
Z39.93-2007
SUSHI
Standard
Revised as
Z39.93-2012
SUSHI
Standard
Revised as
Z39.93-2014
Community
Website
launched
COUNTER SUSHI
Implementation
Profile Published
COUNTER SUSHI
Implementation
Profile updated
2012 2013
SUSHI-Lite
Technical Report
Published
DLF ERMI
Report
2001
PALS
International
Forum on
Usage
10. Brief history of COUNTER and SUSHI
2016 Apr Jun Aug Oct Dec 2017
COUNTER
initiative
launched
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2014 2015 2016
Release 1
Journals &
Databases
Release 2
Journals &
Databases
Release 1
Books
Release 3
Journals &
Databases
Release 4
Journals,
Databases
and Books
SUSHI
initiative
launched
SUSHI Draft
Standard
Released
SUSHI
Standard
Released as
Z39.93-2007
SUSHI
Standard
Revised as
Z39.93-2012
SUSHI
Standard
Revised as
Z39.93-2014
Community
Website
launched
COUNTER SUSHI
Implementation
Profile Published
COUNTER SUSHI
Implementation
Profile updated
2012 2013
SUSHI-Lite
Technical Report
Published
DLF ERMI
Report
2001
PALS
International
Forum on
Usage
11. Release 4 of the COUNTER Code of Practice
Significant features:
• Sessions removed
• Federated and automated searches
• Result Clicks & Record Views added
• SUSHI required
Published 2012, became valid December 31, 2013
12. COUNTER SUSHI Implementation Profile
• Published in 2012 to coincide with
COUNTER Release 4
• Reduce ambiguity and abstraction
• Provides needed guidance to
SUSHI Server developers
• A tool for auditors when checking
for compliance
14. Some statistics on SUSHI Harvests*...
Using SUSHI successfully with more than 100 platforms!
* Source: EBSCO Usage Consolidation SUSHI Harvesting Logs
15. Usus Community Web Site for Usage
• Launched in 2015
• Supported by COUNTER; run by
the community
• A forum for discussions and
feedback
• A place to report problems and
seek answers
• Links to useful resources
http://www.usus.org.uk/
16. SUSHI-Lite: lightweight version of SUSHI (2015)
• Micro-service approach to exchanging
COUNTER data (REST/JSON)
• Much lower barrier to client
development
• Request snippets of usage
• Embed COUNTER stats in other
applications
• Trial use
21. Book Report 7:
Number of Successful Unique Title Requests by Month and Title in a Session
• Unique book views in a session
• Title only gets credited with one “request” regardless how
many chapters, pages, sections viewed in that session
• Replaces BR1 & BR2
• Currently optional
23. New COUNTER Web Site
• Fresh look
• Easier to maintain
• Interactive code-of-
practice
• Searchable list of
compliant vendors
• SUSHI server registry
24. COUNTER Report Validator
• Publically available tool
• Validate SUSHI implementations
• Validate COUNTER reports (XML, TSV and Excel)
• Intended for:
– Vendors to validate their COUNTER implementations prior to audit
and release
– Auditors to reduce the cost and time of the audit
– Librarians to verify reports received and report errors
• Being developed by COUNTER with funding provided by the
EBSCO Foundation
26. R4: Journal Report 1 vs Database Report 1
Database Report breaks out usage
by metric type. Journal (and book)
reports don’t.
Excel version of Journal Report 1 has
totals for PDF and HTML, XML
version breaks them out my month
27. R5: Journal Report 1 (imagined)
In Release 5 all reports would
include the metric type. Ideally with
using the same vocabulary in Excel
as XML
28. COUNTER Release 5
Adding clarity...
• More comprehensive glossary
• Describe how reports and
metrics are used
• Include scenarios
29. COUNTER Release 5
Simplification & Flexibility...
• Consistency between reports
• Fewer “standard” reports
• Include “general” reports
• Introduce filters and attributes
on general reports to replace
feature-specific optional
reports
RELEASE 5
30. COUNTER Release 5
Continuous maintenance...
• Formalize process for
enhancements
• Provide mechanism for
changes to be adopted without
waiting for new standard
RELEASE 5
31. References and further reading
• COUNTER Code of Practice [Release 1], COUNTER, December 2002, url: http://www.projectcounter.org/codeofpractice.pdf
• Electronic Resource Management: The Report of the DLF Initiative, DLF, 2004, published online at: https://old.diglib.org/pubs/dlf102/
• Press Release: NISO Initiative to Standardize Online Usage Statistics Harvesting, NISO, 2005-11-15, published online at:
http://www.niso.org/news/pr/view?item_key=756120eb4a8c7871789809024f18dc81dfdfa345
• SOAP protocol definition, Wikipedia, url: https://simple.wikipedia.org/wiki/SOAP_(protocol)
• eJournal Interface can Influence Usage Statistics: Implications for Libraries, Publishers, and Project COUNTER, Phil Davis, arxiv.org, July 27, 2005.
url: https://arxiv.org/ftp/cs/papers/0602/0602060.pdf
• Release 1 of the COUNTER Code of Practice for Books and Reference Works, COUNTER, March 2006, url:
http://www.projectcounter.org/cop/books/cop_books_ref.pdf
• Release 3 of the COUNTER Code of Practice for Journals and Databases, COUNTER, August 2008, url:
http://www.projectcounter.org/r3/Release3D9.pdf
• Hybrid Open Access Journal definition, Wikipedia, url: https://en.wikipedia.org/wiki/Hybrid_open_access_journal
• COUNTER SUSHI Implementation Profile, NISO SUSHI Standing Committee, August 2012, url:
http://www.niso.org/apps/group_public/download.php/13635/RP-14-2014_COUNTER_SUSHI_IP.pdf
• Continuous Maintenance Procedures Standardized Usage Statistics Harvesting Initiative (SUSHI) Protocol (ANSI/NISO Z39.93), NISO website, url:
http://www.niso.org/workrooms/sushi/continuous_maintenance
• REST definition, Wikipedia, url: https://en.wikipedia.org/wiki/Representational_state_transfer
• JSON definition, Wikipedia, url: https://en.wikipedia.org/wiki/JSON
• SUSHI-Lite: Deploying SUSHI as a Lightweight Protocol for Exchanging Usage via Web Services, NISO SUSHI Standing Committee, July 2016, url:
http://www.niso.org/apps/group_public/download.php/15331/NISO%20TR-06-201x%20SUSHI-Lite%20Protocol%20Draft%20For%20Trial.pdf
Good afternoon.... I am Oliver Pesch and I am here to talk about COUNTER and SUSHI...
I work as product strategist for EBSCO and I also do a lot of work with COUNTER and NISO. For COUNTER I serve on the board of directors, the Executive Committee and am chair of the COUNTER Release 5 Technical Working Group. For NISO I serve on the board of directors and am co-chair of the NISO SUSHI Standing Committee.
Likely most of you are quite familiar with both COUNTER and SUSHI so I will keep the introduction brief so we can discuss recent developments and take a look a head to what is in store for us.
Lets start with a quick background on COUNTER and SUSHI
COUNTER is a Code of Practice for usage reporting – consistent, credible and compatible usage reporting. The code specifies the reports to provide, how they should be formatted, what metrics to include, how transactions should be processed and how the reports are to be delivered. Credibility is enhanced by the requirement for vendors to under go an audit.
Here is the list of “standard” reports – those that must be supplied (when applicable) by the vendor.
These are the ones you are probably most familiar with are... Journal Report 1, Database Report 1 and Book Reports 1 & 2 which provide the basic stats for cost-per-use analysis. And Platform Report 1 is likely the one you use when assembling overall numbers for various surveys and annual reports. We will talk about Book Report 1 & 2 a little later...
SUSHI describes a client/server approach to harvesting COUNTER stats such that an ERM or usage consolidation application, once configured properly, can automatically request and retrieve usage stats on a regular basis (more on that later too).
Here is a timeline that puts this in historical context...
In the late 1990s, the Publishers and Libraries Services group (PALS) began looking into the challenges of usage reporting of online resources. They held an international forum on usage in London in June of 2001 which directly lead to the creation of COUNTER
And the publication of the 1st code of practice in 2003. And as you can see, several releases have followed over the years
For SUSHI, the 2004 DLF ERMI report was the likely its incubator... ERMs were coming on the market and the report acknowledged the promise of COUNTER as a standard for reporting usage... SUSHI came about shortly after as the automated harvesting solution to address the practicality of downloading 100s of usage reports from dozens of content providers.
SUSHI became official ANSI/NISO standard in 2007.
So with that background, lets talk about some developments of note
From the timeline perspective
- the events are COUNTER Release 4, the COUNTER SUSHI Implementation Profile, Usus web site and the SUSHI-Lite technical report...
COUNTER Release 4 had a few impactful changes...
Sessions were removed since federated searching made them an unreliable metric.
Federated searches became “federated and automated” searches. Previously a discovery service or pre-selected multiple-database searches were counted as “regular” searches but counts no longer reflected user action or intent – and database search stats sky-rocketed as a result. With Release 4, “regular” searches are counted hen the user selected the database otherwise they are automated.
And because “searches” were a less reliable metric, result clicks and record views were added to for database assessment reflect user actions related to a given database – a better measure of value.
Plus SUSHI became a requirement for compliance... Though enforcement seemed to be a bit spotty...
SUSHI was a frequent topic of discussion and often not in a good context... Incompatible implementations lead to a lot of frustration and while the standard itself is OK, it is abstract enough to cause confusion to implementers. The COUNTER SUSHI Implementation profile was developed by the SUSHI standing committee to provide specific guidance to developers (and auditors) to remove ambiguity.
While there has been a sense the SUSHI isn’t working, with the efforts of various groups, things are improving...
Lets look at some numbers...
Last February, EBSCO’s SUSHI harvesting was averaging 60% FAILURE rate. Since that time we introduced an updated version of our SUSHI Client...
This coupled with significant improvements on the part of publishers, we have gone from seeing a 60% FAILURE rate in February of 2015 to a 95% SUCCESS rate in February of 2016.
In 2015 the Usus web site was launched. Supported by COUNTER but run by the community, this site provides a forum for discussion and feedback. If you have a question or problem with a SUSHI report or vendor, submit it through the feedback link and the Usus supervisory team will look into it. The track record is pretty good... But normally if a compliance issue is uncovered, the Usus team works with the vendor and the issue is usually fixed quickly.
Also in 2015, the SUSHI Standing Committee published a technical report which described using a current-day micro-service approach to SUSHI. The goal was to lower the barrier to client development as well as offer the option to exchange snippets of usage data so one could, for example, request usage just for one journal and embed that usage in another application. SUSHI-Lite is currently in trial use.
Here is an example of a request – it looks like a URL and it is...
The response is in JSON format and most developers familiar with today’s web development languages can easi;y process this.
And speaking about embedding usage... Imagine a typical renewals screen for a subscription management system... A place where renewal decisions are made...
If we can add use and projected cost-per-use to this screen we have enhanced the user’s workflow by adding key data needed to inform the decision.
Any of you find Book Report 1 and Book Report 2 challenging when trying to compare usage across platforms? Comparing a count of whole book downloads to chapter downloads is virtually meaningless. COUNTER has addressed this with Book Report 7 – currently an optional report that reflects unique book views within a user session. We expect this to become standard in the next release of the code of practice.
OK... Now a quick look ahead
COUNTER will have a new website in a couple weeks. In addition to be more “current”, it will offer interactive browsing of the code of practice and a searchable list of compliant vendors... Plus the SUSHI server registry will be integrated... And searchable by publisher.
Also in the works... COUNTER is developing a COUNTER Report Validator. This will be an online tool that will allow vendors, auditors and librarians to perform quick validation of SUSHI implementations and regular excel COUNTER reports. The goal is to provide a consistent test environment that all implementers can use and thus reduce development time and increase compliance and interoperability.
And then there is release 5 of the COUNTER Code of Practice. You will likely be hearing and reading more about this in the coming weeks and months. The major themes are consistency, clarity, simplification, flexibility all while introducing an continuous maintenance approach.
Consistence... Here is Journal Report 1 and Database Report 1. Database Reports have one row per metric type, whereas, Journal Report 1 is only recording details for Full Text Requests. The Excel version of Journal Report 1 only has reporting period totals for HTML and PDF yet the XML version has these broken out by month.
In release 5 the goal is for consistency between reports and formats. For example, all reports would include usage broken out by metric types and, where practical, the same vocabulary would be used in Excel and XML versions.
Adding clarity through better definitions... Including intended purpose of reports and metric types... And possibly providing scenarios that visually represent a user’s trip though a site and the metrics that result for various user actions.
While it may sound like a contradiction, we are looking to reduce the number of reports and increase flexibility. We would start by getting rid of several of the single-purpose optional reports and replace them with general reports for titles, databases, etc. General reports that can be filtered or configured to meet specific reporting goals. For example. a librarian would be able to select criteria to generate a title report with usage broken out by device type, limited to open access articles...
Finally, the plan is for COUNTER Release 5 to include processes that allow for continuous maintenance so that, much like with SUSHI, changes can be introduced without waiting for a 5 year review of the standard.