Collecting meaningful feedback on Information literacy training: results of a project to evaluate feedback methods - Coles & Perris
•Transferir como PPTX, PDF•
3 gostaram•871 visualizações
Presented at LILAC 2018
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Collecting meaningful feedback on Information literacy training: results of a project to evaluate feedback methods - Coles & Perris
Semelhante a Collecting meaningful feedback on Information literacy training: results of a project to evaluate feedback methods - Coles & Perris (20)
Mais de IL Group (CILIP Information Literacy Group)
Mais de IL Group (CILIP Information Literacy Group) (20)
Último
Último (20)
Collecting meaningful feedback on Information literacy training: results of a project to evaluate feedback methods - Coles & Perris
- 1. Collecting meaningful feedback on information literacy training: results of a project to evaluate feedback methods LILAC 2018, Liverpool 3rd – 6th April Kim Coles – University of Reading Kate Perris – London School of Hygiene and Tropical Medicine
- 3. Review of teaching – and evaluation • 2016/17 a review of information skills teaching included a review of teaching evaluation The aim of this project was to trial alternative methods of feedback collection to evaluate their effectiveness and recommend feedback methods. 3 CC0. Pixabay.
- 4. What is being evaluated? • 1,160 London based students, 3,274 distance learning students • Teaching delivered face-to-face as part of modules, and online via Moodle (VLE) for distance learners • Feedback on information skills teaching has been collected since 2013/14 • Print A4 survey, at the end of each session, completed by hand • Input into MS Access database, report produced each year 4 CC0. Pixabay.
- 5. Why evaluate our teaching? 1. have attendees acquired the skills to meet the learning objectives? 2. do attendees feel more confident in their skills as a result of training? 3. do attendees use the skills acquired in the training in their literature searching? 4. have attendees’ skills improved as a result of training? 5 CC0. Pixabay.
- 6. Feedback methods 6 Short term reflection Long term reflection More complex administration/collection Simple administration/collection *These methods may be used as proxies for evaluation aims, and may be affected by other factors.
- 7. When should we ask for feedback? 7 Cellphone seesaw by Tilemahos Efthimiadis. CC BY-SA. Flickr. [https://flic.kr/p/9e8oRg]
- 8. Which methods did we test? 8 Method Delay Classes Online survey using student bookings system 24 hours Foundation 1 Online survey using Bristol Online Surveys In session IID, IDAC, DH Online survey using Bristol Online Surveys 1 week PHEC, HPPF Online polling using Mentimeter/PollEverywhere In session FRH Print confidence rating question In session Travel Medicine Print One Minute Paper In session Foundation 2 Print survey In session GMH, DrPH Online survey using Bristol Online Surveys In students’ own time* Distance Learning Group 1 Comment and votes using Tricider (snowball-ish) In students’ own time* Distance Learning Group 2 *Feedback was open throughout the online course, students were notified on Moodle and at the end of the course that they could submit feedback
- 9. Response rates 9 *Online feedback using the student bookings system was only requested with a 24 hour delay **Online feedback using the Bristol Online Surveys tool includes feedback collected with a 1 week delay. If only in session Bristol Online Surveys are averaged, the response rate is 68%. All other feedback methods were administered in the session.
- 10. Ease of analysis • How easy was it to collect responses? • How easy was it to input the data? • How easy was it to analyse the data? Online surveys and short surveys (print and online) scored highly in ease of collection and input. Online surveys scored highly on data analysis. 10 Deckchairs at the beach. Sharlon Garland. CC BY. Flickr. https://flic.kr/p/M9EMHb
- 12. What is feedback for? Feedback can serve a number of purposes: • Development: to improve teaching and learning • Appraisal: collect evidence of teacher competence • Accountability: collect evidence of course/programme effectiveness • Innovation: initiate - test/experiment - develop Light, Greg, Dr;Cox, Roy, Dr. 2001., Learning & Teaching in Higher Education. [online]. SAGE Publications Ltd. Available from:<http://www.myilibrary.com?ID=189767> 7 June 2017 12 Handwriting by Anntimony. CC BY. Flickr. https://flic.kr/p/223bHDh
- 13. What questions had we been asking? 13 Student bookings system online survey Bristol Online Surveys online survey Print Survey Print confidence rating Print one minute paper Online poll (PollEverywhere/Mentimeter) Questions on developing teaching Questions on appraising teaching/teacher General questions (eg. cohort) Questions on accountability and effectiveness Questions on innovation
- 15. Implementing the new system of evaluation 3 versions of a feedback questionnaire: • Online Evaluation Form (lshtm.onlinesurveys.ac.uk) • Print Survey • PollEverywhere feedback slides New document including recommendations for use of each survey. 15
- 19. Results and next steps • Which of the three worked best? • What kind of information can we get? • How can this be used to improve teaching? • What other method of evaluation could be useful? 19
- 20. Thank you by Rachel Patterson. CC BY-NC-ND. Flickr. https://flic.kr/p/8h7m69
- 21. Appendix 1: methods of feedback collection Short-term/immediate feedback Written feedback at the end of the class Online feedback at the end of a class One minute paper: What is the most important thing you learned during this session? What is uppermost in your mind now at the end of the session? Plus/delta feedback form: What do you now understand as a result of the session (+) What do you still have questions about (Δ) These are passed around, and students can comment on each other’s comments Reflective triads at the end of sessions 'Snowball' evaluation Ask students to make one positive and one negative statement about the class, place these on the board or pass them round, and ask students to vote on the ones that they agree with Muddiest point: ask students what is still unclear after the session Action plan: ask students to state what activity they will complete as a result of the class/or what they will do differently in their research now 21 alsobytemptationizeCCBYND.Flickr.
- 22. Bibliography • Frutchey, Jim. "Utilizing Google Docs as an Assessment Tool for Academic Reference Librarians." Journal of Library Innovation 3.1 (2012): 148-54. Print. • Gerwitz, Sarah. "Evaluating an Instruction Program with Various Assessment Measures." 42 (2014): 16-33. Print. • Light, Greg, and Roy Cox. Learning & Teaching in Higher Education. United Kingdom: Sage Publications Ltd, 2001. Print. • Meredith, William, and Jessica Mussell. "Amazed, Appreciative, or Ambivalent? Student and Faculty Perceptions of Librarians Embedded in Online Courses." Internet Reference Services Quarterly 19.2 (2014): 89-112. Print. • Nichols, James, Barbara Shaffer, and Karen Shockey. "Changing the Face of Instruction: Is Online of in- Class More Effective?": American Library Association, 2003. 378. Vol. 64. Print. • van Helvoort, A. A. J. "How Adult Students in Information Studies Use a Scoring Rubric for the Development of Their Information Literacy Skills." Journal of Academic Librarianship 38.3 (2012): 165- 71. Print. • Willson, Rebekah. "Independent Searching During One-Shot Information Literacy Instruction Sessions: Is It an Effective Use of Time?" Evidence Based Library & Information Practice 7.4 (2012): 52-67. Print. 22
Notas do Editor
- 1. Polleverywhere icebreaker – using clickable map – where has everyone come from today? 2. Polleverywhere – how confident are you that you are collecting good evaluation on your teaching? Using the happy/angry face scale
- The project took place over the 2016/17 academic year, as part of a larger review of infoskills teaching at the School. In the 2016/17 academic year, the Information Skills team carried out a review of teaching – as well as reviewing the content and delivery of our teaching materials, this provided an opportunity to re-evaluate feedback collection. The aim of this project was to trial alternative methods of feedback collection to evaluate their effectiveness, and to recommend feedback methods which may be preferable, for use in 2017/18 and in the future.
- Our first step was to look at the evaluation which was currently in place, and the teaching that we provided. There are ????? Students at LSHTM - ??? Of which are based in London, and receive face-to-face teaching, and ???? Of which are based anywhere in the world as distance learners. Infoskills teaching is delivered face-to-face as part of modules and via 2 learning packages on Moodle for distance learners. Infoskills teaching includes introductory or Foundation teaching at the start of term, and then more advanced search skills for projects. All LSHTM students – F2F or DL – receive the same content via either format. Some additional teaching in the form of systematic review searching is part of the curriculum in other modules – such as the Reviewing the Literature module – and some teaching is also offered to RDS and staff. Feedback on teaching had been collected for a number of years, in the form of a print A4 survey passed out at the end of each session and completed by hand. This was input into an MS access database, and a report on the comments and general ratings of the class was produced each year. So we actually had a fairly good idea of how the teaching had been received. But there were issues e.g. remembering to take the feedback forms to the class, time to collect feedback at the end of a session, how reliable is feedback collected immediately after a 3-hour class? Could we make the labour-intensive job of input easier? And would this help us to make producing a report easier, and more timely?
- Fairly quickly we started talking about why teaching was evaluated and what we wanted to get out of this evaluation. In previous years recommendations had been determined from the teaching feedback, and one of the Library & Archives Service’s KPIs was a certain percentage of satisfaction with teaching. But could more be done? The 4 objectives on the screen were identified as potential aims for evaluating infoskills teaching – but we also felt that 2 of them could not be assessed using the kinds of feedback methods that the project had the scope to cover. We wanted our students to acquire the skills they needed to complete their assignments and projects, and confidently find reliable, up to date, medical information. But we were aware that one class doesn’t always translate into a change in search behaviour. We split our objectives up, and decided that for the scope of this project we were going to evaluate short-term feedback methods – but we identified alternatives that could be used to assess more long-term changes to searching behaviour as well.
- Through pooling ideas from feedback methods that we had either used previously, come across as part of teaching and learning ourselves, we quickly formed a very long list of potential methods. I found it helpful to try to categorise these according to their potential uses, and their ease of use. As mentioned previously, we decided to focus on the short term reflection – and how this could work better for us as teachers. Is everyone comfortable with the different methods of feedback gathering in this slide?
- We also considered when we should request feedback. We felt that it was likely (and through reading the literature) that a delay in requesting feedback would lead to reduced responses. But we also wanted to recognise that after a period of time, students might be able to reflect a little more on the teaching and whether it had been useful. It sometimes feels like feedback is a balance between getting representative numbers, and getting reflective responses.
- We decided to allocate at least one method to each session (2 where possible). This table shows the status of each session at the end of the academic year/project period. Each of the acronyms in the classes field refers to a particular class. We weren’t able to test all the methods that we had wanted, due to attendance in sessions or issues with arranging a class. But we felt that we had a good spread of different methods and some delays in questioning which would allow us to get an idea of the responses received, and how effective each of these methods might be as a feedback collection method. The existing print survey was converted into an online format, that was administered both using Bristol Online Surveys tool, and the in-house student bookings system, which has an evaluation function built-in. We also included the distance learners – who could already respond via a Moodle survey, or contact us via email, which some did. We set up a new survey using BOS, and tried out a more interactive form of commenting and providing feedback using Tricider.
- This graph shows the overall responses for the different methods of feedback. It includes delays and so it’s important to note that without the delays the picture does change slightly, with online surveying coming up at 68% response rate if done at the end of class. Most of our feedback was administered in the session, or as part of the session. Print surveys, one minute papers, and online surveys were all administered after the class had finished. The confidence rating and online polling were part of the session e.g. a question about confidence was asked at the beginning of the class, and then followed up at the end. The student bookings system has an inbuilt questionnaire feature which can be used to automatically send a feedback questionnaire to students who have attended a class. This was used after the Foundation 1 classes in October 2016, and sent on a day after the class. The Bristol Online Surveys (BOS) institutional subscription was used to create an online survey which students could access in class or sent on via email. This was used in sessions, and also sent onto students via email 1 week after the class Online polling via Mentimeter and PollEverywhere was also used in sessions to collect feedback. Polls can be embedded in presentations, or a link can be provided to students to access via a PC, laptop, or mobile device, and the administrator of the feedback can view and export feedback after the class. Written feedback was also used in the form of a quick confidence rating (pre- and post-class), and a One Minute Paper. In some classes a print survey was also used to collect more detailed feedback. Each of these methods was used at the end of a class. Our distance learners had a very, very low response rate. And so we couldn’t include them in the results of this project. So we concluded that delays lead to very few responses, which would lead to misrepresented results, and so collecting feedback as part of or immediately after a session was preferable. We were a little surprised that the print feedback forms had a higher response rate than those online, especially as these included the same questions, and were administered at the same time. But it seemed clear that asking in the session was preferable and so the student bookings system, which only allowed for feedback collection with a delay, wasn’t going to help. Shorter methods had higher feedback rates, but not as much as expected - our longest print survey was 13 questions, and this was completed almost as much as the One-Minute paper.
- But what could we do with the information afterwards? How did our various methods make analysis easier? We recognised that collecting, collating, and analysing feedback can also cause bottlenecks in fully understanding how teaching was received, and how it is working. Time spent inputting data could be spent considering the results, and so we also reviewed the ways in which the data collected could then be used. We rated each method according to ease of collecting responses, ease of data input once responses had been collected, and ease of viewing/analysing data collected. This is very personal to our own working methods, but it was a useful exercise in thinking about the balance between collection and use of the information! The Bristol Online Survey, PollEverywhere, and short print questions scored highest on ease of collection (these methods were also all used as part of or at the end of a class). Both online surveys scored highest on easy of data input, as data input was not required. Of the print methods tested, the shorter methods (confidence rating and one minute paper) were also scored highest in this area. The Bristol Online Survey and print One Minute Paper scored highly on ease of viewing/analysis, it was easier to view in the system, and to export the data. The polls automatically saved answers that were easy to view, and could be exported and compared with other cohorts. Again, the student bookings system was more difficult to extract data, as each survey was attached to each individual class. BOS allowed us to use one survey for all classes, and include a question on the cohort/class attended so that we could filter by class in future as necessary. Requesting feedback at the end of a session increases the response rate. Print feedback obtains a higher response rate than online feedback, even in the session. Print surveys require data input, unlike online surveys. Shorter data collection methods were easier to analyse but may not collect information in all relevant areas. It is possible to improve the poll feedback method that was used in this year’s evaluation: polls can be embedded throughout a class presentation to collect feedback on various areas of the class eg. including questions on impact and future development of the class. Some of the shorter methods of feedback collection, although requiring less administration, do not collect feedback on all areas which may be beneficial to improving future teaching, or demonstrating the impact of a class, and so may not be appropriate for wider use. Another issue encountered during the year was that the different methods of collection asked different questions, and so it is not possible to reliably compare feedback across cohorts. It is suggested that for the next academic year fewer collection methods using similar questions are compared and trialled, using the information gathered this year to assess suitability of each method for each cohort.
- 3. Polleverywhere – what do you want to find out from your evaluation? 1 minute only for this so that we can move on!
- Another issue encountered during the year was that the different methods of collection asked different questions, and so it is not possible to reliably compare feedback across cohorts. We wanted to collect feedback on the following areas: have attendees acquired the skills to meet the learning objectives? do attendees feel more confident in their skills as a result of training? We could use the feedback to asses what we were doing well in the class, and to improve what we did. We could also use it to collect evidence that classes were well-received, which helps the Library to promote classes and recruit more academics and students to infoskills teaching.
- Most of the questions in previous feedback methods related to accountability – collecting evidence of course/class effectiveness. A large number of questions also related to development – to improve teaching and learning. As a team, it is felt that the questions related to accountability and development were the most useful, as we would like to use the feedback methods to gather information on the impact of the classes, and also places where the classes can be improved. We produced a set of 12 questions, 3 asked about development of learning, 3 appraised the class/teacher, 3 measured effectiveness, 2 provided opportunities to innovate, and 1 was a general question about the cohort the students belonged to. Focussing on accountability and development enabled us to look closely at: the impact of the classes areas where the classes can be improved.
- Once we had decided on questions, and analysed the results of the project, we settled on one feedback questionnaire, and created 3 versions of it that can be used in teaching.
- JISC BOS provides dashboard for responses. Can see all results to compare across the academic year or filter. Online evaluation form – public URL – useful if your attendees have internet access as part of the class, or you can email them after the class to complete a form (response rates are greatly reduced with a delay in feedback gathering) you do not have time to input or analyse responses after the class
- Need to scan and analyse by hand – and Print survey – useful if: your attendees do not have internet access during the class (e.g. Foundation classes in Week 1) you do not have email addresses of attendees to collect responses after the class you are able to input/analyse responses after the class
- PollEverywhere – useful if you are using a powerpoint presentation your attendees have internet access as part of the class - you would like to embed feedback as part of class discussion