This paper will begin with an overview of digital libraries and usability and a brief discussion of recent trends in usability testing of digital libraries. That is followed by a brief discussion of continuous quality improvement of a digital library, specifically the implementation of a follow-up survey to be completed by the general population who visit a publicly accessed digital library. Then a follow-up instrument is introduced.

1. Kathryn Brockmeier
kathryn.brockmeier@nebraska.gov
Lincoln, Nebraska
Use of a follow-up survey for improvement of a digital library
This paper will begin with an overview of digital libraries and usability and a brief discussion
of recent trends in usability testing of digital libraries. That is followed by a brief discussion of
continuous quality improvement of a digital library, specifically the implementation of a follow-
up survey to be completed by the general population who visit a publicly accessed digital library.
Then a follow-up instrument is introduced.
Definition of a digital library
A digital library is a repository of digital content. Digital libraries are typically accessed
online, using computers or other electronic devices. Librarians and information technology staff
work together to create an environment where the user, whether it be the public, researchers, or
employees, can easily search for. Definitions of a digital library abound. According to the Digital
Library Federation (1998):
Digital libraries are organizations that provide the resources, including the
specialized staff, to select, structure, offer intellectual access to, interpret,
distribute, preserve the integrity of, and ensure the persistence over time of
collections of digital works so that they are readily and economically available for
use by a defined community or set of communities.
Definition of usability
Usability testing is formal testing of a website whereby participants complete tasks and may
be observed by the test administrator or tracked electronically. Users may be asked to respond to
questions about these tasks. One of the most widely cited definitions of usability is found in the
International Standards Organization’s (ISO) 9241-11, which purports that usability is “the
1

2. extent to which a product can be used by specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a specified context of use” (p. 2)
There is a body of literature which studies usability of library websites (for example,
Cervone, 2014; Letnikova, 2008). Since then, a growing number of studies have examined the
usability of digital libraries (Comeaux, 2008; Dickson, 2006; Jeng, 2005; Jeng, 2006; Joo and
Lee, 2010; Kahl and Williams, 2006; Kani-Zabihi, Ghinea, and Chen, 2006; Koohang, 2004;
Koohang, 2004; Zimmerman and Paschal, 2009). Much of the literature is dedicated to usability
testing of academic digital libraries. One exception is the Zimmerman and Paschal (2009) study,
which included testing of a non-academic digital library.
Users and subjects
In many instances, the subjects were students or faculty at an academic institution, and the
digital library was of an academic nature. Very little literature is devoted to the usability of
digital libraries of cultural heritage institutions or non-academic libraries. Dickson (2006),
however, did include members of the general public in that case study.
Continuous quality improvement
Usability testing and instrument creation in the literature have overwhelmingly focused on
stages of creation of the digital library. But there is a need to monitor the digital library for
continuous quality improvement. In the National Information Standards Organization’s (NISO)
(2007) paper A framework of guidance for building good digital collections, Initiatives Principle
4 states that “a good digital initiative has an evaluation component.” (p. 91) Specifically:
While output measures such as the number of items digitized can be useful, recent
emphasis is on outcome assessment, which is concerned with how people,
collections, organizations, and systems have been affected by the project. The
2

3. evaluation plan should emphasize the importance of an ongoing two-way dialogue
with key stakeholder communities. (p. 91)
As Cervone (2014) states, “It is not possible to know if a website is truly meeting the needs
of patrons if there is no on-going program of assessment.” (p. 11) Cervone (2014) adds, “As a
mechanism for assessment and quality improvement, evidence-based practice is critical.” (p.12)
Surveys
Surveys, also known as questionnaires, are one way to capture the impressions of those using
a digital library. A questionnaire can serve to gather data on several dimensions of use. There are
various good reasons conducting a survey: “Although survey methods might not produce
detailed strategies directly applicable to system improvements, they are nevertheless useful in
gathering data from relatively large samples and diagnosing the current status of usability in
general.” (Joo and Lee, 2010, p. 534) NISO (2007) states that surveys are a good way to measure
inputs and outputs.
A questionnaire is not necessarily usability testing. It is more a method to gather empirical
data about a user’s impressions of the digital library interface and content. Most of these studies
focus on testing via observable tasks and answers to questions related to those tasks. Some
studies include a follow-up survey to gauge perceptions of the digital library. Several assess the
methodology of usability testing on digital collections, including the creation of a standardized
instrument grounded in empirical research. Some questions were open-ended while others were
built on a Likert scale.
There are several frameworks that propose underlying attributes, criteria, or constructs which
contribute to ascertaining usability. Nielsen’s (1993) framework is one of the most widely cited
usability models. It introduces five attributes—learnability, efficiency, memorability, errors, and
3

4. satisfaction—as subordinate elements of usability. Frameworks that specify the usefulness of a
digital library include:
• Kani-Zabihi, Ghinea, and Chen (2006): Functionality, interface/usability, and content
• Dickson (2006): Usefulness, effectiveness, learnability, and likability
• Jeng (2006): Satisfaction, broken down into the areas of ease of use, organization of
information, terminology and labeling, visual appearance, content, and mistake recovery
• Koohang (2004): Simplicity, comfort, user friendliness, control, readability, information
adequacy/task match, navigability, recognition, access time, relevancy, consistency, and
visual presentation
• Joo and Lee (2010): Efficiency, effectiveness, satisfaction, learnability
Survey instrument questions
The questions in the proposed instrument are drawn and modified from previous studies as well
as generated to address several underlying constructs of usability: efficiency, effectiveness,
satisfaction, and learnability. Wording and terminology are selected to be understandable by the
lifelong learner, as opposed to the academician or college student. This question bank could be
tested for validity and reliability and the list of questions narrowed down.
1. Overall, how satisfied are you with your use of this digital library today?
• Very satisfied
• Satisfied
• Neither satisfied nor dissatisfied
• Dissatisfied
• Very dissatisfied
2. How did your use of this digital library meet your expectations?
• Better than I expected
• Just as I expected
• Worse than I expected
3. How easy or difficult was it for you to navigate this digital library?
• Very easy
• Easy
• Difficult
• Very difficult
4

5. 4. Were you able to find what you were looking for today?
• Yes
• No
5. Did you ever feel like you got lost on this site?
• Yes
• No
6. Did you browse the collection by subject?
• Yes
• No
• I don’t know
7. How helpful were the search feature?
• Very helpful
• Helpful
• Not very helpful
• Not at all helpful
• I didn’t use this feature
8. How useful were the help features?
• Very helpful
• Helpful
• Not very helpful
• Not at all helpful
• I didn’t use this feature
9. Are you a (check all that apply):
• Teacher
• Student
• Researcher
• Lifelong learner
• Other (please describe) ________________________________
10. Comments about your visit today: ______________________________
In summary, there is a need for institutions to make an effort toward planning for evaluation
of their digital libraries. While academic libraries have made strides in this area, there is room for
cultural heritage and information service institutions to do the same.
5

6. References
Cervone, F. (2014). Evidence-based practice and web usability assessment. OCLC Systems &
Services, 30(1). 11-14.
Comeaux, D. J. (2008). Usability studies and user-centered design in digital libraries. Journal of
Web Librarianship, 2(2/3), 457-475.
Dickson, M. (2006). CONTENTdm digital collection management software and end-user
efficacy. Journal of Web Librarianship, 2(2-3), 339-379.
Digital Library Federation. (1998). A working definition of digital library [1998]. Retrieved from
http://old.diglib.org/about/dldefinition.htm
International Standard Organization. (1997). ISO 9241-11: Ergonomic requirements for office
work with visual display terminals (VDTs), Part 11: Guidance on usability specification
and measures.
Jeng, J. (2005). What is usability in the context of the digital library and how can it be measured?
Information Technology & Libraries, 24(2), 47-56.
Jeng, J. (2006). Usability of the digital library: An evaluation model (Unpublished doctoral
dissertation). Rutgers University, Brunswick, N.J. Retrieved from ProQuest.
Joo, S. & Lee, J. (2010). Measuring the usability of academic digital libraries: Instrument
development and validation. The Electronic Library, 29(4), 523-537.
Kahl, C. M. & Williams, S. C. (2006). Accessing digital libraries: A study of ARL members’
digital projects. Journal of Academic Librarianship, 32(4), 364-369.
Kani-Zabihi, E., Ghinea, G. & Chen, S. Y. (2006). Digital libraries: what do users want? Online
Information Review, 30(4), 395-412.
Koohang, A. (2004). Development and validation of an instrument for assessing users’ views
about the usability of digital libraries. Issues in Informing Science & Information
Technology, 1, 55-63.
Koohang, A. (2004). Students’ perceptions toward the use of the digital library in weekly web-
based distance learning assignments portion of a hybrid programme. British Journal of
Educational Technology, 35(5), 617-626.
Letnikova, G. (2008). Developing a standardized list of questions for the usability testing of an
academic library web site. Journal of Web Librarianship, 2(2/3), 381-415.
National Information Standards Organization (NISO). (2007). A framework of guidance for
building good digital collections. Retrieved from
http://www.niso.org/publications/rp/framework3.pdf
Nielsen, J. (1993). Usability engineering. Boston: Academic Press.
Zimmerman, D. & Paschal, D. (2009). An exploratory usability evaluation of Colorado State
University Libraries’ digital collections and the Western Waters Digital Library web
sites. Journal of Academic Librarianship, 35(3), 227-240.
6

7. References
Cervone, F. (2014). Evidence-based practice and web usability assessment. OCLC Systems &
Services, 30(1). 11-14.
Comeaux, D. J. (2008). Usability studies and user-centered design in digital libraries. Journal of
Web Librarianship, 2(2/3), 457-475.
Dickson, M. (2006). CONTENTdm digital collection management software and end-user
efficacy. Journal of Web Librarianship, 2(2-3), 339-379.
Digital Library Federation. (1998). A working definition of digital library [1998]. Retrieved from
http://old.diglib.org/about/dldefinition.htm
International Standard Organization. (1997). ISO 9241-11: Ergonomic requirements for office
work with visual display terminals (VDTs), Part 11: Guidance on usability specification
and measures.
Jeng, J. (2005). What is usability in the context of the digital library and how can it be measured?
Information Technology & Libraries, 24(2), 47-56.
Jeng, J. (2006). Usability of the digital library: An evaluation model (Unpublished doctoral
dissertation). Rutgers University, Brunswick, N.J. Retrieved from ProQuest.
Joo, S. & Lee, J. (2010). Measuring the usability of academic digital libraries: Instrument
development and validation. The Electronic Library, 29(4), 523-537.
Kahl, C. M. & Williams, S. C. (2006). Accessing digital libraries: A study of ARL members’
digital projects. Journal of Academic Librarianship, 32(4), 364-369.
Kani-Zabihi, E., Ghinea, G. & Chen, S. Y. (2006). Digital libraries: what do users want? Online
Information Review, 30(4), 395-412.
Koohang, A. (2004). Development and validation of an instrument for assessing users’ views
about the usability of digital libraries. Issues in Informing Science & Information
Technology, 1, 55-63.
Koohang, A. (2004). Students’ perceptions toward the use of the digital library in weekly web-
based distance learning assignments portion of a hybrid programme. British Journal of
Educational Technology, 35(5), 617-626.
Letnikova, G. (2008). Developing a standardized list of questions for the usability testing of an
academic library web site. Journal of Web Librarianship, 2(2/3), 381-415.
National Information Standards Organization (NISO). (2007). A framework of guidance for
building good digital collections. Retrieved from
http://www.niso.org/publications/rp/framework3.pdf
Nielsen, J. (1993). Usability engineering. Boston: Academic Press.
Zimmerman, D. & Paschal, D. (2009). An exploratory usability evaluation of Colorado State
University Libraries’ digital collections and the Western Waters Digital Library web
sites. Journal of Academic Librarianship, 35(3), 227-240.
6