Making the connections: assessment, training, progression and performance. Harrison & Newton
•Download as PPT, PDF•
1 like•184 views
Presented at LILAC 2007
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Making the connections: assessment, training, progression and performance. Harrison & Newton
Similar to Making the connections: assessment, training, progression and performance. Harrison & Newton (20)
More from IL Group (CILIP Information Literacy Group)
More from IL Group (CILIP Information Literacy Group) (20)
Recently uploaded
Recently uploaded (20)
Making the connections: assessment, training, progression and performance. Harrison & Newton
- 1. University Library Making the connections: assessment, training, progression and performance Dr A. Harrison: Institute of Psychological Sciences A. Newton: Leeds University Library
- 2. Why this project? BSc Psychology (Hons) A progressive and integrated degree programme that encourages students to become more autonomous thinkers and practitioners of Psychology Culminating in an independent, empirical project reporting on a substantial piece of research
- 3. Do we need Information Literate students? Quality Assurance Agency Generic Skills Retrieve and organise information effectively:- books, journals, computer and internet sources Handle primary source material critically Become more independent and pragmatic as learners
- 4. The story so far Lilac 2005 Second year students are more information literate than first year students First year students have a strong preference for using the internet to source information Second year students use the Library Catalogue as their primary literature searching tool Both groups use the Library Catalogue ineffectively
- 5. Current study Aims: • Assess the reliability of the data produced by the information literacy questionnaire • Provide detailed information about the progression of information literacy skills acquisition by students • Assess the relevance of information literacy skills to academic performance
- 6. A reliable tool? Comparing results No significant differences between the performance of the two first year groups (2005-06/2006-07): t = 0.909, df = 382 p=0.364 No significant differences between the performance of the two second year groups (2005-06/2006-07): t =1.58, df = 383, p>0.115 Year of entry 2005 2006 First year Students Mean % correct +/- (SED) 48.94 (9.89) 47.96 (11.67) Second Year Students Mean % correct +/- (SED) 61.69 (12.80) 59.63 (13.22)
- 7. Tracking progression Second year students do progress in Information Literacy 0 10 20 30 40 50 60 70 80 90 100 Within Group Comparison (N=112) Betw een Group Comparison (N=805) Mean%Correct First Year Second Year
- 8. Progression Areas of significant progression Search strategies Locating and accessing information Organising and applying information Areas of less progression Recognising an information need Identifying ways to fill an information need Comparing and evaluating information
- 9. IL progression Second year students are: Better at keyword searches Better users of Boolean logic Search Strategy 0 10 20 30 40 50 60 70 80 90 100 First Year Second Year Students Mean%ofStudentsAnswering Correctly
- 10. IL progression Second year students are far more competent at referencing Neither group have sufficient understanding of copyright Organise and Apply 0 10 20 30 40 50 60 70 80 90 100 First Year Second Year Students Mean%ofStudentsAnswering Correctly
- 11. IL progression Both groups fail to use search facilities effectively More second year students understand the usefulness of a bibliography in a printed book Locate And Access 0 10 20 30 40 50 60 70 80 90 100 First Year Second Year Students Mean%ofStudentsAnswering Correctly
- 12. Less progression Good performance by both groups: Very good at acknowledging gaps in their information Appropriate strategies for dealing with information overload Room for improvement: Lack skepticism First year students use of online encyclopedias Recognise Need 0 10 20 30 40 50 60 70 80 90 100 First Year Second Year Students Mean%ofStudentsAnswering Correctly
- 13. Less progression Good performance in both groups: Understand purpose of journal article abstracts Over ½ correctly identified differences between websites and peer reviewed journal articles Understand peer review process Compare and Evaluate 0 10 20 30 40 50 60 70 80 90 100 First Year Second Year Students Mean%ofStudentsAnswering Correctly
- 14. Less progression Poor performance by both groups: First year students source most information from internet search engines Second year students are more discerning Second year students look to the internet for recent academic information Review articles: both groups unaware of appropriate use Identify Ways to Fill Need 0 10 20 30 40 50 60 70 80 90 100 First Year Second Year Students
- 15. Academic performance First year Students In general, Information Literacy ability of students entering Higher Education is strongly related to their academic performance in year one semester one at University Mean Academic Performance: (r =0.349, n =130, p<0.001) HIGHLY SIGNIFICANT Multiple choice examination performance: (r = 0.326, n=130, P<0.001) HIGHLY SIGNIFICANT Practical report writing performance: (r = 0.208, n = 130, p<0.05) SIGNIFICANT Essay writing performance: (r = 0.152, n = 130, P=0.084) NOT SIGNIFICANT
- 16. Academic performance Second year students After some initial Information Literacy training in Higher Education the strong relationship between IL ability and academic performance is enhanced Mean Academic Performance: (r =0.355, n =118, p<0.001) HIGHLY SIGNIFICANT Multiple choice examination performance: (r = 0.290, n=118, P=0.001) HIGHLY SIGNIFICANT Practical report writing performance: (r = 0.269, n = 118, p=0.03) SIGNIFICANT Essay writing performance: (r = 0.181, n = 118, P<0.05) SIGNIFICANT
- 17. Summary Reliability Data produced by the questionnaire is reliable across different cohorts of students and the LILAC 2005 data suggesting a progression of IL ability from year 1 to year 2 has been replicated Progression The Questionnaire now provides detailed information about student abilities including areas of substantial or no progression Academic performance Information Literacy is strongly related to academic performance
- 18. Newton-Harrison 7 pillars of evidence based IL training Pre-training Recognisein ofyourstude Identifyways needsofyou Constructat programme Deliveryoftra Compare&ev studentabilit progression Analysestud performance Synthesised evidence-bas Post-training Pre-training
- 19. Future plans 3rd year student data to be obtained Re-examine IL in the Psychology curriculum Further investigation into academic performance: A-level subjects Dissertation results (3rd year students) Final degree results
Editor's Notes
- A new Information Literacy training programme for psychology undergraduate students at the University of Leeds was launched in 2003. In order to ensure that the training programme addressed the needs of the students we created a tool based on the 7 pillars model of IL and the Quebec study ( IL Qnn ) that would allow us to assess their IL skills ability, the progression of their skills acquisition and therefore the success of our training programme.
- Although the necessity of IL skills is implicit in almost every degree programme the QAA has to some degree made these explicit in their benchmark statements
- Substantial revision of the Qnn (summer 2005) to assess a range of abilities related to each of the six key skill areas and to ground it specifically in the discipline of psychology, thus encouraging student engagement in the study. Now used the revised Qnn for two consecutive years and today will present the data from these two years highlighting the usefulness of regularly assessing the students skills as a means of assessing the effectiveness of your training and the progression of student skill acquisition
- a) ensuring that given a similar population of respondents (for example two groups of first year students) similar data is obtained at a different time (t-tests L105 vs L106 & L2o5 vs L2o6) b) Compare the performance of first and second year students using much larger group sizes thus making the statistics more powerful. By this we mean that we now have a large sample of first and second year respondents so we can have some confidence that our data accurately represents these populations. This analysis was also run to check that the data presented in LILAC 2005 was reliable and accurate 2. having run the study with the new Qnn for two consecutive years we can follow the students who entered in 2005 through the first year of acquisition of these skills, and thus compare the performance of the same students at the start of their first and second years of study. 3. And finally we have analysed whether the IL ability of students (as measured by our Qnn) is related to their academic performance as measured by a variety of assessment methods.
- We have now run the Qnn for two consecutive years and so we have data from two first and second year cohorts of students. To assess the reliability of the Qnn we compared the results from these two groups and found that:
- we can now follow a single group of students (N=112) from entry into our degree programme through to their second year of study. In this way we can now report data relating to the progression of skill acquisition of a single cohort of students (those that entered our programme in September 2005). A similar pattern to the between group comparison (N=805) previously mentioned is seen with second year students performing better on the Qnn we can now use the detailed data generated by the Qnn regarding student performance in the different skills areas to give us a clear picture of what skills they are learning and those that they are not or put in a different way what are we good at teaching and what are we not good at teaching OR how is our training programming failing to meet our students need
- This analysis was conducted on the whole cohort of respondents from first year in 2005 and second year in 2006 Doesn’t mean to say that less progression means they were useless to begin with – might mean there is little room for improvement in the first place.
- (however their understanding of the use of truncation is atrocious (26.8% got it correct) despite having been trained in this and used it as part of an online tutorial for PSYCINFO GRAPH first yearsecond year Mean44.8661.52
- Copyright, think they can print or p/copy as much as they want to (– indicates a more general attitude towards information i.e: they expect it to be free?) GRAPH first yearsecond Year Mean48.8066.70
- (both fail- they expect the library catalogue to behave like google- that is that all search boxes are holistic in that they will access all information , rather than being specific to particular datasets, for example they are particularly bad at knowing what part of a reference you use to find a journal article- L2= 17.5% correct) (however, there is still room for improvement in this group) Confusion of index with bibliography Not all progression is necessarily IMPRESSIVE GRAPH first yearsecond Year Mean24.3641.73
- Good at taking action over inclusive search results – deal with information overload Seem to trust information without giving sufficient consideration to issues of bias and fail to seek out independent corroborative evidence (50%) – question relating to government information – Ofcom study on teenagers and media literacy confirms this. Second year students more likely to seek help from textbooks when they don’t understand concepts. GRAPH first yearsecond Year Mean68.7072.20
- - Both groups know the purpose of journal article abstracts - More than half of the students in both groups understand the difference in quality/ validity of information contained in websites in comparison with peer reviewed J articles - A similar proportion of the students have an understanding of the peer review process BUT a third of second year students don’t know the quality differences between these sources of information or the peer review process- so why read journals??? GRAPH first yearsecond Year Mean71.8675.43
- Consistent with our LILAC 2005 presentation - First year students heavily dependent upon and show a strong preference towards inappropriately sourcing all information from the internet search engines - second years students are becoming more discerning about the sources of their information but many are still look to the internet for the most recent academic information - Most students in both groups are unaware of the appropriate use of review articles GRAPH first yearsecond Year Mean38.2442.68
- In Leeds, at the end of each 11 week semester students are examined
- IL 1 measure weeek1 semester 1 Yr 1 was compared to semester 1 yr1 academic perf IL 2 measured week 1 semester 1 yr 2 was compared to academic perf at end of semester 2 yr1 (closest assessment period and before they received any further training To assess whether each students Il ability was related in any way to their academic performance corelative analysis was conducted between these two performance indicators.
- IL 1 measure weeek1 semester 1 Yr 1 was compared to semester 1 yr1 academic perf IL 2 measured week 1 semester 1 yr 2 was compared to academic perf at end of semester 2 yr1 (closest assessment period and before they received any further training To assess whether each students Il ability was related in any way to their academic performance correlative analysis was conducted between these two performance indicators.
- Pre-training Recognise need: There is no point in creating a training programme without knowing what the needs of your students are- they may surprise you good OR bad. Remember not only do you need to assess the abilities of new incoming students, but also more advanced students e.g. those that you may assume have certain abilities. (our data clearly demonstrates that even after specific explicit training our second year students have not acquired certain skills- much to our dismay!!) Identify: There are now many different teaching methods used to address the training issues associated with IL training (online tutorials, library treasure hunts, classroom activities), but also remember that people are resources, we feel that the success of this work is largely attributable to the consistent and direct interaction between library staff and academic staff. This interaction is critical if you want to follow best practice guidelines and integrate your skills training into the academic curriculum Construct: Only when you know the needs of the students can you provide them with a, timely and progressive programme of training Post-training: After delivery there must be some assessment of the abilities of the students and analysis of this data should be used to inform future training thus producing evidence-based training
- There is much still to explore, and we’re not done yet!