VIP Model Call Girls Koregaon Park ( Pune ) Call ON 8005736733 Starting From ...
Evaluating a new electronic service using mixed methods
1. Evaluating a new electronic service
using mixed methods
Presentation to QQML, 2012
Denise Turner and Sue Myer
Teesside University
2. Outline of Session
• Background
• Introduction to LibGuides
• Evaluation stage
• Opportunity for your questions and feedback
3. Background
• Wide variety of resources available
overwhelming for students
• Subject guides just list of links
4.
5.
6.
7. LibGuides
• Creates a one stop shop for a subject
• Interactive multimedia
• Integrated help at the point of need
• Customisable
• Linked to modules in the VLE
8. Questioning LibGuides
• How to justify an ongoing subscription?
• Are they being used?
• What do people think about them?
– Academic staff
– Librarians
– Students
9. LibGuides Usage – top 25
Forensic Science
Graphics
Interior Architecture and Interior Design
Media
Youth Studies September 2011- April 2012
Sociology
Midwifery and Women's Health Total hits = 161,465
Teaching in the Lifelong Learning Sector
Radiography, Imaging and Ultrasound
Accounting and Finance
Computer Games and Animation
English
Physiotherapy
Law
Paramedic Science
Education
Criminology
History
Early Childhood Studies
Sport and Exercise
Computing
Occupational Therapy
Nursing
Psychology
Business and Management
0 2000 4000 6000 8000 10000 12000 14000 16000
10. Variation in usage
• Some guides not being used heavily
• Top guide:
– Business - 13,471 hits
• 81st guide:
– Food & Consumer Safety – 44 hits
• Some content areas within guides not being
used
11. Business Guide - Sections
Page Total
Welcome 4977
Finding Journal Articles 3519
Journals 1285
Electronic books 1116
Finding Books 815
More Resources 760
Referencing 327
Useful Websites 254
Statistics 157
Getting Started 140
Off Campus Access 98
Darlington 23
Totals 13471
12. Google Analytics
• Only introduced May 2012
• Who are the users?
– Initial results – majority coming from Teesside
• How are users getting to site?
– Nearly 50% users going directly to libguides
• How long are users spending on the site?
– Average visit 3 mins 38 secs
13. • Less than 2% on a
mobile
• Only 3.5% from
search engine
• 55% referred to
site
• 95% of referrals
come from Library
website
14. • Excellent site!
• I will be using for my assignments
• Useful informative and easy
access what more could a
student want?
• Really handy
BUT
• Less cluttered
• Search box could be more
specific
15. Feedback Form Limitations
• Only 24% added comments
• Collating individual emails
• Who are they from?
• Too many feedback forms on the site
– confusion with feedback for information literacy
sessions
– “more time to be shown”
16. Academic Staff
“I just wanted to say how good the Lib guides are. They
really help my teaching and final year projects :)”
• Learning and Teaching Conference
1. Should LibGuides be extended to include study
skills?
2. How can we involve academic staff?
18. Answers to: How can we involve
academic staff?
• Who decided how many sub divisions eg SCM 2 guides not
enough? Sohsc many but overlap between them.
• Who decides subject for guide areas can we influence this eg
research methods or generic health not just nursing if new
research doesn't cover this?
• Name link
• Build in area for staff suggestions
19. Librarians
• Focus group with librarians:
– What are the positive aspects of LibGuides?
– What are the negative aspects of LibGuides?
– What do the statistics tell us?
– What should we test with students?
22. What do statistics tell us?
• Perhaps reduce number of subject specific guides eg Nursing, Engineering
• Common pattern in section usage across the 6 most popular guides for
each School
• Can see impact of training sessions in usage statistics
• Finding journal articles 2nd highest on each - wording makes it easy to
find?
• Are headings meaningful?
• More resources hides expensive databases
• Low book usage
• Usage stats for databases don't reflect actual usage - students accessing
from elsewhere or bookmarking
23. Students
• Have the guides lived up to our expectations?
• Can students find the information that they
need?
• Will we find some of problems that other
research has found?
24. How far have we got?
• Usability study protocol developed
– based on University of Washington Libraries, USA
• Technical set up sorted out
– using Camtasia in designated room
– Student workers
• Tasks agreed
• Studies to be carried out this week
25. Tasks for usability study
1. Find a libguide for Forensic Science
2. Find the electronic book:
Dingwall, G. (2006) Alcohol and crime. Willan.
3. Locate the following journal article and open
the full text:
Low, B. and Laffey, D. (2011)"Is Twitter for the Birds? Using Twitter to
Enhance Student Learning in a Marketing Course." Journal of
marketing education 33(2) pp.183-192.
26. Tasks for usability study
4. Find the database Proquest Nursing Journals
and open the factsheet.
5. Find out why Katy Demoily has recently been
in a newspaper.
27. Conclusion
• Evaluation has developed from original project
plan
• Pleased with usage
• Evaluative approach could be used for other
resources and services
• Work in progress!
– overview of the sessionThis session will introduce you to a project that we have been running at Teesside University to introduce a new electronic service at Teesside University. In September we introduced LibGuides a hosted web based services to support information literacy and so this presentation will be about the various methods we have been using to evaluate this new service and reflecting on our approach. So we will start by explaining the background and what LibGuides are and how they have been developed. We will share with you how we have tried to gain information from stakeholders and what information we have gained so far (eg on their usage) and what we plan to do next.There will be an opportunity for your questions in the end.
Denise – Before we implemented Libguides, we had links to all of the resources we had for a subject in subject guides, but these were very static, and the structure of our web site meant that the help for the resources was separate from the actual resources. The guides themselves were just a list of links and the headings for the sections of the subject guides had be agreed as each subject area needed to use the same headings.
So for example this is what the business subject guide – looked like. As I’ve said just a list of links to various types of resources. There was no focus on a task a student might want to perform they needed to understand what online databases via Search It meant – there was information below to explain a little more but do people read it?Students got to their subject guides from the navigation on the left – because of the timing of the project teaching material had already been designed telling students to click on the link to Subject Guides and colleagues were reluctant to replace this with a LibGuide link and so for this year we kept both the original Subject Guides and introduced the new LibGuides.
BusinessLibGuideThis is what the new Business LibGuide looks like. For anyone not familiar with LibGuides the new service uses a library content management system to present information tailored to a specific discipline. It is very easy to create guides and all staff in my team were involved where in the past only a small number of the team would have had the skills needed to create a webpage like this. We aimed to bring together all of the most useful resources for a subject, along with instructions, tutorials and details of the academic librarians supporting that subject - in effect creating a one stop shop.We have one of these replacing each of our original subject guides. Initally we created a template which was used by all subjects, this speeded up conversion to the new guides and meant that core information is only input once and then cascaded to all LibGuides. We are now working on developing them to be more discipline specific. The structure isTab-based, organised by a mixture of different formats of information and tasks. This is the welcome screen: our strategy is to push relevant info at students through the VLE rather than to expect them to navigate our web site. Each Blackboard module will link to the relevant subj guide. We are using RSS feeds to add the latest news to this page and a calendar of events that updates to show the latest month . This is also the opportunity to introduce the subject team.
Through areas like Finding books we can push key texts to students. Through the e-books – area we can promote the eversion of heavily used print titles , as these can be fairly hidden from students. In the section we are looking at now we can point out major journals collections and online for that discipline together with factsheets, tutorials etc. shows students how to reference this type of material – in a relevant style for their school.
Denise –So in summary, some of the positive aspects of libguides.Customisable so able to tailor them to the needs of a subject in consultation with academic staff
Denise We have been monitoring usage of LibGuides since they were introduced and take up has been good. 161,465 hits so far in comparison to the Old subject Guides which over the same time period last year = had 41,747 hits. this years usage of the old guides has dropped to just 10,725 a drop of just over 74%. (74.31%)
Statistics from Sep-April.Show graph of under-used guides – take to librarians meeting?Compare a few well-used guides – top 5 – any trends?
Limitations of libguides statistics – no idea of the path that users taking or time spent or how reached the site – so implemented Google Analytics. Any results?
One of the early decisions we made on planning the evaluation of LibGuides was to use a feedback form. This is a facility that LibGuides gives you and building a form is very easy. We implemented a feedback form on every LibGuide. Asking users to rate the LibGuide out of 5 based on how useful it was. Space was also available to add comments. Despite high usage we have only received 85 feedback forms from the site. None from the Business School who had the most used guide! Most respondents scored the guides highly and based on the score we could have got we achieved almost maximum scores. Some students also left comments and many of these were also positive. Plus there were some useful suggestions about how we could improve guides.
However, only 24% of the returned forms included comments and where the score was low there was often no comment at all which was frustrating. Unlike other methods we had used for forms in the past like Survey Monkey of Google Docs the responses came as individual emails and so even with the low response it was time consuming to collate the feedback. We didn’t know who was returning the forms and so We thought this would give us a good opportunity to obtain feedback on the Libguides – but it hasn’t proved as useful as we had initially hoped. For next year, we may not abandon feedback but will consider using our own survey form which will mean feedback will automatically be collated.
Swap to Sue – going now from general feedback to our evaluation with specific user groups, beginning with academic staff.Introduced guides to them before we launched them to students – but only interested staff attended. Feedback from those staff was very positive. Subject librarians publicised the guides with their schools – again feedback was positive, but we don’t know whether the staff then promoted the guides with students.We gave a presentation on LibGuides to our Teesside Learning and Teaching Conference this year. We asked whether libguides should be extended to include “study skills” and How can we involve academic staff?
When we asked whether libguides should be extended to include “study skills” these were the areas that were mentioned.When we looked at the feedback from the L&T Conference even though only one group mentioned Dissertation, there was lots of feedback that could be encompassed in a guide to writing a dissertation eg report writing, time management – we have a separate project working on this. So we found this feedback useful.
We didn’t find these responses particularly useful – quite random statements that didn’t always link to the questions, but maybe that this was due more to the limitations of the conference session:Limitations of methodology eg range of attendees (also included learning technologists and ICT), difficulty of engaging them with actual question within short timescalePut into groups and asked as a group to write on post it's and struggled to get feedback so 5 mins is not enough. Could have asked for longer session but an hour too long?In hindsight should have asked them to start individually and then discuss(NHS not able to access due to firewalls. Problems with social media and ICT departments.)
Focus group earlier this month.
Strong agreement on positives – especially that they’re easy to create, also easy to use, brings everything together, and looks good
More varied comments – concerns over info overload, content being hidden, “dumping ground” (negative side of easy to create). Also finding a balance between shared template and freedom to ‘do your own thing’
A lot of these thoughts were what we’d already decided before the session – but it worked well to allow colleagues to see this for themselves from the evidence rather than just tell them – eg have fewer guides.Questions about whether the headings are meaningful especially More Resources were addressed by the usability study
Students are the most important user group. When we thought about what we wanted to learn from them, these were our key questions:Expectations – look back at advantages that we mentioned at the start of this presentation and seeing if reality matchesCan students find what they need? Do students find them overwhelming? (info overload concern of librarians)Shared problems – reference:LibGuides Usability TestingCustomizing a Product to Work for Your UsersRachel Hungerford, Lauren Ray, Christine Tawatao, Jennifer WardUniversity of Washington Libraries – Seattle, Washington USA https://digital.lib.washington.edu/dspace/bitstream/handle/1773/17101/UWLibGuidesUsability-2010LAC-tawatao.pdfTesting carried out in 2009.Also research from:
Usability protocol – based on research from Uni of Washington, but also Nielsen’s web usability advice and our own experience of testing our web site several years ago.Also informed by all our evaluation up to this point – aspects that we particularly want to know about such as whether students find the tabs and sub-tabs easy to use Specific tasks - As well as 5 Tasks, X and Os exercise when students indicate useful and less useful parts of pages, also more general questions about how use libguides – up to 1 hour in total.
Tasks – start with easy (basic) question2. and 3 - Key tasks that want students to be able to do – find expensive resources –ebook and journal article
Two tasks based on specific concerns:4. Do students realise what the resource icons do (for factsheet) – do they understand that online databases are within Finding Journal Articles5. Newspaper database beneath More Resources – does this hide them? (particular concern of librarians)Early days – can only report on 2 students experience – but demonstrated keenness to use search box rather than navigate the site. Also just as Google Analytics is giving us more than we expected, these usability studies are also providing us with much richer data.
Work in progress – not yet completed usability study but also needs to be iterative process (make changes, test, make changes ….)
20 mins here - Sue - We are at an early stage and so value both your questions but also your feedback