Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
•
6 gostaram•1,916 visualizações
This document summarizes a large scale longitudinal study examining the effect of discovery systems on online journal usage. The study analyzed journal usage data from 24 academic libraries before and after implementing a discovery service. It found that while some publishers saw an increase in usage and some saw a decrease, the change in usage varied significantly depending on both the discovery service and the individual library. The study utilized ANOVA models to determine that discovery service, library, and the interaction between discovery service and publisher were significant predictors of changes in journal usage. The results indicate the effect of discovery systems on usage is complex and depends on multiple factors.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
Semelhante a Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage (20)
Mais de Jason Price, PhD
Mais de Jason Price, PhD (20)
Último
Último (20)
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage
- 1. Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage Charleston Conference November 7, 2013 Michael Levine-Clark, University of Denver John McDonald, University of Southern California Jason Price, SCELC Consortium
- 2. “…a steep increase in full text downloads and link resolver click‐ throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/
- 4. Does implementation of a discovery service impact journal usage?
- 6. Web-scale discovery services • Single source for finding information – Books – Articles – Local content • Metadata and/or full text • Content is pre-indexed and/or pre-harvested • Single fast search ILS ILS Publisher Publisher Metadata Metadata MLA MLA Bibliography Bibliography Institutional Institutional Repository Repository HathiTrust HathiTrust Discovery Service Discovery Service
- 7. An assumption • At any given institution, given a relatively stable user base, the total search effort will remain roughly the same. – X students will have Y assignments and Z hours per day to search – X faculty will publish Y papers and have Z hours per day to search
- 8. Discovery services Will take up an increasing amount of a finite time for searching Will draw users from other (more or less efficient) search tools Will alter the overall productivity of searches (users will find more or less) Will alter the overall efficiency of users (users will access more or less full-text)
- 9. Prior studies • Some studies have indicated substantial increases in usage after Discovery implementation – Descriptive statistics only – Single institution studies only • Some publishers report decreased usage of content – Anecdotal, may affect some and not others
- 10. Data collection • List of libraries with discovery services > Searched on lib-web-cats • Surveyed Libraries > Discovery service Implemented > Implementation Date (month/year) > Search box location > Marketing effort • 149 Libraries Gave Approval > 24 libraries selected for this phase > 6 for each of the 4 major discovery services
- 11. Library demographics • 20 US, 1 each from UK, AUS, NZ, CA • 10 ARL Libraries included • WorldCat book holdings > Average: 1,114,193 > Median: 1,044,153 > High: 2,665,796 > Low: 298,365 • Implementation dates: > 2010 (3), 2011 (19), 2012 (2)
- 13. Dataset • 24 Libraries • 4 Discovery services • 6 Publishers • 9,206 Journals • 159,278 Observations • 141,048 Usable Observations
- 14. Methodology Compared COUNTER JR1 total full text article views for the 12 months before vs 12 months after implementation date Year 1 Year 2 Included implementation month in Year 1 to ensure that both periods included an entire academic year
- 15. Collections notes o Excluded journals that did not have 24 months of COUNTER reporting o Limited ability to control for changes in aggregator, backfile access, or expanded holdings o Outliers removed from analysis
- 22. General trends • Variation by institution within each discovery service • Variation by publisher within each discovery service • Some publishers saw overall net increase, while some experienced a decrease in usage
- 24. Goals of our inferential statistics Determine whether observed differences are significant or resulted from chance effects Determine which of the three factors (i.e. library, publisher, discovery service) contribute to determining differences in usage change at the journal level Start with an exploratory analysis and end with a comprehensive model
- 25. ANOVA - Analyzing the data Observation = Fit + Residual Change In = Library x + Publisher y + Disc Svc z + Residual Err usage +17 = (+2) + (-3) + (+10) + (+8) After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1
- 26. ANOVA – F Ratio Tests whether the means for levels within a factor are distinguishable from each other Average variability due to the factor F-ratio = --------------------------------------------------Average variability due to chance error So, when F ≈ 1, means are not distinguishable, when F is > 1, there are real differences among some means
- 27. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
- 28. Does usage change vary across libraries? Overall Average = 8.5 Institution (sorted by Mean Change)
- 29. Does usage change vary across publishers? Overall Average = 8.9 Publisher (sorted by Mean Change)
- 30. Does usage change vary across discovery services? Overall Average = 8.9
- 31. Does the affect of discovery service differ across publishers?
- 32. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
- 33. Do the discovery service means differ in the 2 way model? 15.0 12.3 4.5 3.7 Publishers (distinguished by color)
- 35. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
- 36. Do publisher means differ significantly in the two way model? 23.8 6.8-9.5 -3.9 Publishers (distinguished by color)
- 37. Does the affect of discovery service differ across publishers? Publishers (distinguished by color)
- 38. Publishers (distinguished by color)
- 39. Full Model – including Discovery service, Publisher, and Library
- 40. ANOVA Model including all three factors
- 41. Results - Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions? Discovery Service – Yes Publisher – No Library – Yes Differential discovery service effect by publisher – Yes Differential library effect by publisher -- Yes
- 42. Interpretations & Conclusions Analyzing usage is a complex task No discovery service increased or decreased usage across all libraries and/or all publishers > Discovery service and publisher as variables on their own were significant predictors of usage change > Interaction of Discovery service & Publisher was significant > Some control needed for no discovery service and for size of institution. > >
- 43. A plethora of pending possible pursuits • Design & test for effects of: –Aggregator full text availability –Institution Size / Enrollment Profile –Publisher Size –Journal Subject –Overall usage trends (Requires Disc Srvc ‘control’) –Configuration options in Discovery services • Follow-up presentation at UKSG (April 2014) –Including Control group & Additional libraries –Add Additional variables & further analysis
- 44. Questions michael.levine-clark@du.edu | johndmcd@usc.edu | jason@scelc.org
Notas do Editor
- Selecting the libraries: Maximized number of libraries balancing the number and size of libraries representing each discovery service and keeping implementation dates within a year or two of each other
- Mention that there were only 3 institutions + publisher groupings that we had to leave out Usable observations = journals at institutions that had counter report values for individ
- Explain concept of outliers and why they are removed. Z-Score on the Y axis is calculated as the number of standard deviations from the mean. Of the 141,048 obs, we eliminated only ~100 outliers.
- Although we were asked to keep the identity of the publishers confidential, we have used a consistent color scheme to identify results that pertain to each publisher
- Summary of all journals from a publisher to the libraries having that discovery service. Many of the bars include total journal observations in excess of 10,000 while some of those, particularly for the smallest two publishers (in green an orange) include between 500-1000 observations.
- Analysis of Variance (or ANOVA ) allows us to determine whether the dffferences we observed are significant The analysis Breaks down the influences on each observation into the effect of levels of the factor of interest (I.e. Lib, Pub, Disc) and error The numbers show an analysis of one observation – each value expresses the difference from the mean of 1 level (Lib x, Pub Y, and Disc Srvc Z) as well as the portion of the value that is not determined by these factors (i.e. the residual error) If we imagine the hundreds of observations for each combination, we can understand that the size of the residual error relative to the value determines whether the effects are significant predictors of the observed values
- Determines whether means (or averages) of each level within a factor are distinguishable from each other or put another way it assesses the likelihood that journal change values sampled from different levels of a factor (say libraries) are actually estimating the same population… or statistically different populations
- So, in response to our first question: Does usage change vary across libraries? We see the 24 libraries sorted by mean change along the x axis and mean change in usage plus or minus two standard errors on the y axis Standard errors are a measure of the variability around each individual library In general, when 2xSE bars overlap, those means are not distinguishable The overall average change was 8.5…. And our F-ratio of about 32 tells us that institution alone is a significant predictor of mean change in usage after discovery service implementation.
- Whenever the p value (shown in the significance column) is less than 0.05, it indicates that we can reject the null hypothesis that there is no differences among levels of the factor (in this case, libraries) But for our single factor anovas, this ignores the impact of different discovery services and journal publishers on mean change in usage
- Click The grand mean for change in usage across publishers is 8.9 One publisher appears to have a mean change that isnt distinguishable from zero, whereas… And the significant F value and non overlapping error bars suggest that the mean change varied across publishers, BUT As we will see, when we add the other factors we’ll find that these differences are actually explained by discovery service and institution effects rather than publisher differences
- This data shows that the mean usage increase was positive for all discovery services, …. Although we cant distinguish these from industry wide increases since we didn’t examine usage change in libraries that did NOT implement a discovery service So when we just look at discovery srvc & ignore the variation due to publisher or library we do see differences
- When we include both Discovery Service and Publisher in a Two way model, We can ask whether we can detect differences when we take both Discovery Means & Publisher means into account In addition to asking whether we can detect a difference across discovery service alone and publisher alone The two way model addresses whether the impact of discovery service is equivalent for each publisher We can think of it as asking whether these lines are parallel (statistically) or whether they cross
- These are the same data in the previous slide separated into panels by discovery service. The following slides will step through testing for an effect of discovery service, publisher, and their interaction
- Do the discovery service means we see here differ significantly in the two way model?
- Do the discovery service means we see here differ significantly in the two way model? No– they do not.
- Do the publisher means we see here differ significantly in the two way model?
- Do the publisher means we see here differ significantly in the two way model? No–they do not.
- Does the affect of discovery service differ across publishers?
- Does the affect of discovery service differ across publishers? Yes, it does. Statistically these lines are not parallel.
- Mixed Nested Parially-crossed model Acknowledges that each library can only implement one discovery tool Takes all three variables into account in the same test.
- To reiterate the full model results,