1. Stephen J. Stose June 19, 2010 1
IST 676: Assignment 2
In what follows is a general outline for a method of evaluating digital libraries. This paper has
two goals: 1) to refine and continue to develop effective criteria for the evaluation of digital
libraries, and 2) the use of these criteria in the practice of evaluating two specific digital libraries.
Two federated digital libraries are evaluated across five criteria of evaluation. The criteria were
adapted from multiple sources, most principally from that of Saracevic (2005) and Choudhury
(2002). While the criteria presented here do not necessarily follow the structure of these previous
authors, they do attempt to incorporate and re-organize the same multiple dimensions generally
accepted as essential. These dimensions are listed in bold, each followed by a series of questions
that attempt to manifest each concept. They do not constitute orthogonal dimensions. Yet they do
seek to represent psychologically distinct experiences of a typical user’s episode with the digital
library resource. For this reason the criteria attempts to model this user flow. That is to say that
upon entry, the user has 1) a first impression, followed by 2) a judgment of its aesthetics and
overall design, which incorporates signposts (links etc) that allow/disallow its 3) usability; entry
to the digital objects themselves is facilitated or constrained by the exposure and effectiveness of
the sites 4) taxonomic system, which lead to the 5) digital content itself a user hopefully can
contextualize and learn from. Such a description of an episode is of course a user-centric ideal,
and exposes the author’s own preferences towards good digital library design (i.e., how he
believes a site should foster navigation and resource discovery). However, it also serves as a
model for further research, given that these propositions regarding user-centric episodes and the
criteria by which they should be judged are themselves testable and hence falsifiable.
University of Wyoming Digital Initiative The Arizona Memory Project
http://digital.uwyo.edu/ http://azmemory.lib.az.us/
The overall quality The front page immediately invites you A very brief time here assures you that
of organization: Is into its various “recently digitized” one interface is the rule. The front page
the content collections. Navigating from the main quickly divulges a lot, in a friendly
immediately interface into separate collection manner. It has teasers of some of its
accessible and interfaces is realistic, but there is seldom a content (usually collections
inviting with clarity clear way back to the collection options. highlighted), to attract the user inside. It
in its presentation? Many collections are federated within one boasts of teacher resources that, within
Does its overall space (LUNA), which provides coherency seconds, show you the availability of
structure serve its if the user is lucky to land there. Others lesson plans for teachers. You see
stated goals? What lead one quite astray into university quickly that this is some kind of
impression does the programs with uncertain connection to federated system with many institutions,
first 15 seconds of this digital initiative. This fragmentation and a quick click on ‘browse’ or
entry leave? is disorienting, a fact the ‘about’ section ‘search’ reveals much of the metadata at
failed to allay. The search feature on the one’s fingertips. An extremely attractive
front page was both ineffective and start.
misleading. The portal’s overall quality is
therefore very low.
Design: Is the site The main page attractively The design is simple, yet effective.
attractive and presented/exposed the various collections There are few items in the menu, and
simple to visually available up front. If lucky, most these are repeated with short
navigate? Does the collections lead to a separate LUNA explanations within the main content.
design assist the workspace (a DAM), which is very More specialized features (‘help’, ‘my

2. Stephen J. Stose June 19, 2010 2
IST 676: Assignment 2
user in orienting its attractive, simple and customizable, and favorites’, and ‘my preferences’) are
contents? Is the federates all searching for collections less conspicuous, but available
design stable and within its domain. If unlucky, other immediately. The site uses only one
psychologically collections lead to very distinct spaces interface throughout, and a fluid width
coherent and this can be very disorienting and allows use of your entire screen. The
throughout, or does organizationally distinct. The front page is site is attractive to navigate, mostly
it require continual simple and visually attractive, but the because it is simple with links that
visual and menu options usually just offer a series of resolve with the information they boast.
conceptual re- links that lead users into a separate web The ‘help’ and ‘about’ are long html
orientation as one interface, thereby visually and documents, many times unadvised in
navigates? psychologically placing them again in a information architecture, but a quick
new space. Thus, the design lacks an menu upfront immediately breaks up
umbrella space and visual guideposts that this content by scrolling you to each
organize collection subspaces and ensure question listed in its menu. Very
navigational cohesion. effective, even if a trifle bland.
Usability: Is the If lucky to land inside LUNA, where This is an incredibly simple and
resource easy to many of the collections reside, the users effective digital library, which speaks
use and effortless to must learn one mere interface in order to well of OCLC’s CONTENTdm, its
navigate? Do the navigate across its collections. Links underlying architecture. This is a
links resolve in resolve very quickly and the navigational federated digital library, and the links
expected places, menu is stable, but a user does need to illustrate this effectively. Within one
and is the user experiment a while to learn how they click of the front page you can read
correctly oriented function. It is not immediately clear when about the contributing institutions, and
to retrace this navigation is within or between within two clicks (‘browse>’select a
space? Are collections, but this price of federated collection from the list’) you can view
navigational cohesion is resolved soon enough. There the items just from that institution’s
signposts stable is very effective user support, but using collection. One click on ‘favorites’ or
and informative? Is this resource effectively is not effortless, ‘my preferences’ shows the user how
the user aware of given its power to afford users many the interface can be changed to
the various features organization powers: media groups, accommodate personalization for
available with presentations, API embedding, and other improved usage. The help section is
sufficient guidance options given registration. The site may available, but given the ease of this
in their operation? be too powerful at the cost of ordinary interface, almost superfluous, I dare
and quick presentational capacity for say. A user has no issue at all finding
more ordinary user groups. what s/he needs, or just browsing.
System and The main taxonomy is the collection-level A taxonomy of categories is
metadata: Does a categorization. The “Search” feature on immediately available by clicking on
taxonomy of the main page is useless. If within LUNA, ‘browse’. While this list is quite sparse
categories enable the advanced search has excellent (broken into ‘collections’, ‘topics’,
effective resource dropdown fields with extensive metadata ‘formats, and ‘time periods’ only), it
discovery? Is the and Boolian options, and allows federated broadness is indicative of the
metadata exposed searching within or amongst all abstraction needed to properly federate
to users to enable collections. Simple searching reveals the collections of over 90 institutions.
effective search? faceted options in a sidebar, expanding or This is a wonderful way to get users
Do queries return refining the results. It is difficult to find into the content immediately, and listed
presentable results the “browse by category” section in order beside each thumbnail is basic ‘subject’
digestible to human to immediately learn about the items and ‘description’ metadata, which
users? Are queries available, but when found the “what”, allows for quick digestion of its

3. Stephen J. Stose June 19, 2010 3
IST 676: Assignment 2
handled efficiently? “who”, and “when” categories do provide relevancy before going further. Facets
Are there faceted a needed conceptual framework for the in the sidebar allow narrowing the
options for collection’s holdings that might serve the fields. When choosing an image, much
continued user better if constantly available in a more metadata unfolds beneath the
refinement? sidebar. Queries provide results with image itself, or when doing an advance
various grid and object resize options, search. Queries are handled
with item-level summaries available immediately, such that you forget you
below or upon hovering over each item. are amongst 67,000 digital objects.
Digital content: This single-interface federation (in The links ‘my preferences’ or ‘my
Does informative LUNA) is excellent for combining high- favorites’ allows you to change the
descriptive (e.g., quality items across/within the collection, screen defaults (sort order, background
historical) material but contains little collection-level color, grid/list options), such that the
contextualize the information, leaving little to learn about content itself appears according to each
digital objects? Are the history and context of each collection. user’s desire. You learn a lot very
the objects Attempts to return to the main portal for quickly regardless of the format, as
informatively collection-level guidance are difficult and beneath each thumbnail are effective
labeled at the item- otherwise unhelpful, as “about” section descriptors, and beneath each image
and collection- discusses issues more relevant to itself the metadata is co-referenced with
level, to make it university members, and the “exhibits” links to that reveal all the objects within
easy to learn? Is and other menu items show links that lead that ‘category’. Browsing is fun, which
the content a high to unfamiliar space. Individual items are implies you are learning and engaged in
digital quality, and well documented in the left sidebar with a the content. Checkboxes beside each
can users complete listing of technical, descriptive, object allows you to collect content
manipulate (e.g., and administrative metadata. Users can (‘my favorites’). Image size is limited,
collate, zoom, print, add/collate items to a workspace or but you can request larger sizes. The
itemize) the presentation space, zoom and compare, if best feature is the item-level to
objects? they care to learn these features (and collection-level coherency. Regardless
register). This plethora of multimedia of how you came upon an item, it is co-
features overshadows the storytelling and referenced to information regarding its
contextualized nature a digital library also collection. Thus, one learns about the
needs to represent. item and its context easily and
effectively.
Is the digital The site has potential, once loose ends are This digital library is fantastic, one of
library mission collected. The site fails to serve as an the best federated sources I have used,
supported? umbrella portal except that the first page and very worthy of emulation. It
allows users to enter specific collections informs on the item- and collection-
(with little way back, however). The level, integrates these seamlessly within
LUNA federated interface is an excellent one interface, while still recognizing the
multimedia tool that exhibits great control collection has a separate institutional
over item-level metadata, but it fails to provenance. The mission is completely
incorporate collection-level information, supported. Besides a few design issues
and probably sacrifices historical regarding attractiveness (menu fonts)
storytelling of the items within each and fuzzy header images, it is top-class
collection for multimedia options ordinary in terms of usability.
users may find difficult.

4. Stephen J. Stose June 19, 2010 4
IST 676: Assignment 2
A brief conclusion is warranted. Obviously the Wyoming Digital Initiative is under construction.
This I know, as I know one of the directors (iSchool Graduate Ben Goldman). For this reason,
the analysis may not be completely fair. However, given that I mostly concentrated on the
federated LUNA system, some words of direct comparison may be granted. While the power and
complexity of the LUNA system is quickly apparent, these benefits have as costs the joy of
browsing I experienced within the Arizona Memory Project. Too much complexity may instill
only confusion, and given that most visitors just wish to browse and perhaps do basic educational
research, the power afforded within the LUNA system is perhaps a trifle unnecessary, if not at
odds with its educational purpose. In my opinion, if and when a user requires high-level
multimedia capabilities, these should be secondary to a digital library’s public and educational
(i.e., information quality should be first). The LUNA system does raise the bar for digital library
architecture. This power, however, should be secondary to a good experience.
Saracevic, Tefko (2005). How were digital libraries evaluated? Presented at Libraries in the Digital Age
(LIDA), Dubrovnik and Mljet, Crotia, May 30-June.
Choudhury, G.S.; Hobbs, B.; M Lorie, Flores, N.E. (2002). A Framework for Evaluating Digital Library
Service. D-Lib Magazine July/August 2002. Volume 8 Number 7/8