NDF Barcamp, Hamilton City Library, Hamilton, New Zealand

1. How effectively are crowdsourcing
websites supporting volunteer
participation and quality contribution?
Donelle McKinley
PhD candidate, School of Information Management
Victoria University of Wellington
NDF Barcamp, Hamilton, 21 June 2013
www.digitalglam.org @donellemckinley

2. Crowdsourcing: a short version
Crowdsourcing outsources tasks
traditionally performed by specific
individuals to a group of people or
community through an open call (Howe,
2009).

3. Crowdsourcing: a long version
“Crowdsourcing is a type of participative online activity in which an
individual, an institution, a non-profit organization, or company
proposes to a group of individuals of varying knowledge,
heterogeneity, and number, via a flexible open call, the voluntary
undertaking of a task. The undertaking of the task, of variable
complexity and modularity, and in which the crowd should
participate bringing their work, money, knowledge and/or
experience, always entails mutual benefit. The user will receive the
satisfaction of a given type of need, be it economic, social
recognition, self-esteem, or the development of individual skills,
while the crowdsourcer will obtain and utilize to their advantage
that what the user has brought to the venture, whose form will
depend on the type of activity undertaken.”
(Estellés-Arolas & González-Ladrón-de-Guevara, 2012)

5. Potential benefits of crowdsourcing
for the institution
O Continue tradition of volunteerism
O Market and stimulate interest in collections
O Signal the institution's openess and approachability
O Achieve goals otherwise too costly and labour-intensive
O Better reflect the diversity of visitors
O Tap into expertise outside the institution
O Engage visitors in new ways
O Raise the profile of research
O Demonstrate relevance
O Enable new research questions to be explored

6. Common motivations for volunteer participation
O The size of the challenge
O The necessity for volunteer contribution
O Collaboration with prestigious institutions
O Contribution to research
O Education
O Mental stimulation
O Being part of a community
O Personal research interests
O Enhancing a resource from which they will benefit

7. The potential of the crowd
Digitalkoot (National Library of Finland)
O First 51 days of the project
O 31, 816 visitors to the site
O 15% participated
Transcribe Bentham (University College London)
O First 6 months of the project
O 1,207 visitors registered to participate
O 21% participated

8. Three common scenarios
The website either:
O Follows the structure of the underlying
technology or the organisation
O Adheres to familiar conventions
O Is the product of personal preference
Garrett, J. (2011). The Elements of User Experience: User-centered
design for the web. USA: Aiga/New Riders.

9. How effectively does your website…?
O Define its objectives
O Reflect visitor motivations
O Align visitor motivations with relevant
incentives
O Minimize sources of frustration and
concern

10. How are they impacting on the effectiveness
of your website?
O Content
O Language
O Readability
O Website navigation
O Arrangement of page elements
O Consistency
O Visual appearance
O Page load speed
O Number and complexity of processes to complete
the desired action

11. O Prototype: an early sample or model built to test a concept
or process
O Pilot: a small-scale preliminary experiment conducted to
evaluate feasibility, time, cost, adverse events, and improve
design prior to the launch of a full-scale project
O Soft-launch: the release of a website to a limited audience,
in order to (beta) test and tweak a design before being
launched to a wider audience
O Beta testing: user testing by a limited audience to ensure
the website/software has few faults or bugs, and gather
user feedback
O Optimization: increasing the percentage of visitors that
fulfill the objective of a webpage or website

12. User evaluation: A different kind of user
engagement
O Focus groups
O Surveys
O Usability testing
O Ongoing feedback channels

13. Analytics:
How is website design impacting on…?
O The number of online visitors
O Time spent on site
O The number of online visitors who register to
participate
O The number of online visitors who actually
participate
O The number of abandoned and completed tasks
O The number of return visitors

14. OCR text correction: Trove, National Library of Australia

15. Evaluation of the Trove crowdsourcing
user interface involved:
O Asking potential volunteers to comment on the prototype
O Sitting potential volunteers in front of the computer
screen and asking them how they would complete the
task
O A soft launch and beta testing
O Gathering feedback on the beta version from over 600
users over the course of five months, via a survey, an
online contact form, direct observation of user activity,
analytics, online comments, and direct contact with
users via email and phone

17. Evaluation of the Transcribe Bentham task
interface involved:
O Beta testing
O A user survey
O Website analytics
O Analysis of user interaction statistics
O Comparisons with studies on
crowdsourcing volunteer behaviour

18. UK Reading Experience Database (UK-RED)

19. Evaluation of the UK-RED task interface
involved:
O Heuristic evaluation
O A survey of current and potential
contributors
O Comparison with other crowdsourcing
task interfaces
O Comparison with recommended practice
as outlined in crowdsourcing and HCI
literature

20. Usability and functionality requirements for
a NZ-RED task interface
O Minimize user effort
O Support integration of the task with research
processes
O Enable new visitors and contributors to understand
what the task involves quickly and easily
O Support accurate and controlled data entry
O Be easy to use for people reasonably confident with
the Web
O Support flexible, structured data entry
O Support bilingual data entry

21. Challenges impacting on evaluation
O Funding
O Expertise
O Organisational culture

22. What could a more effective website mean
for the project?
O More online visitors participate
O Tasks are completed more efficiently
O Tasks are completed with greater
accuracy
O The task is more enjoyable
O Volunteers participate more often
O The project is more cost-effective
O More volunteers are willing to participate
in future projects

23. Thanks!
For references and other great reads visit
http://www.digitalglam.org/crowdsourcing/books/
Presentation and slides will be available at
http://www.digitalglam.org/crowdsourcing/talks/
For crowdsourcing research updates follow
www.digitalglam.org @donellemckinley