Loex2012
•Transferir como PPTX, PDF•
0 gostou•258 visualizações
Assessment of a Team-Based Embedded Librarianship model
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Loex2012
Semelhante a Loex2012 (20)
Último
Último (20)
Loex2012
- 1. Assessment of a Team-Based Embedded Librarianship Model Leo Lo, Research & Development Librarian Jason Coleman, Undergraduate & Community Services Librarian http://www.slideshare.net/JasonColeman1/
- 3. Information Literacy is Served Jason Leo Total Posts Made 33 13 46 Posts Read 902 856 1758 E-mails 1 1 2 answered Consultations 1 0 1 IM Chats 10 0 10 Online helps Libguide, 3 2 assignment Libguide, 5 created assignment tips tips documents assignment tips documents documents
- 4. Cooking & Prep Time Jason Leo Total Time Spent 656 minutes 201 minutes 857 minutes during class Time Spent in 420 minutes 240 minutes 660 minutes preparation Total 1076 minutes 441 minutes 1517 minutes
- 5. Consumption Aug. 2011 Jan. 2012 Total Students 53 52 105 Students who 8 7 13 posted questions to AAL thread Total posts on AAL 56 30 86 thread Total AAL posts read 1184 951 2135 Avg # AAL posts 22 18 20 read/student Libguide n/a 407 page hits 407 page hits IM chats 56 10 66
- 6. Was it Nutritious? 1. Our impact on Pre-test to Post-test improvement 2. Our impact on Overall grade 3. Feedback
- 7. Pre-Test Question 1 Find an estimate of the current population of Beijing, China. What is the estimate? a. Where did you find this estimate? b. Do you think it is accurate? Why or why not?
- 8. Pre-Test Question 2 Find a current street map of Beijing, China. a. Where did you find this map? b. What process did you use to find it?
- 9. Pre-Test Question 4 How many tourists visited Beijing in 2010? a. Where did you find this figure? b. Do you think it is accurate? Why or why not?
- 10. Pre-Test Question 5 Find and cite (APA or MLA) a credible article on one of the ethnic minorities in China. Also, in two to three sentences, justify the credibility/authoritativeness of your source.
- 11. Post-Test Question 1 Find an estimate of the current population of Buenos Aires, Argentina. What is the estimate? a. Where did you find this estimate? b. Do you think it is accurate? Why or why not?
- 12. Post-Test Question 2 Find a current street map of Buenos Aires, Argentina. a. Where did you find this map? b. What process did you use to find it?
- 13. Post-Test Question 4 How many tourists visited Buenos Aires, Argentina in 2010? a. Where did you find this figure? b. Do you think it is accurate? Why or why not?
- 14. Post-Test Question 5 Find and cite (APA or MLA) a credible article on the status of women in Argentina. Also, in two to three sentences, justify the credibility/authoritativeness of your source.
- 15. Pre-Test to Post-Test Study Sample: • Treatment group (n=10) Average # of posts to AAL thread: 2.5 Average # of posts to all threads: 13.9 • Matched control group (n=10) Average # of posts to AAL thread: 0 Average # of posts to all threads: 13.9
- 16. Pre-Test to Post-Test Scoring Rubric Instructions: For each ACRL Standard circle Beginning, Proficient, or Advanced. If you are not able to make an assessment, write n/a ACRL Standard Beginning Proficient Advanced 1. Determine the The research query is The research query is The research query is Extent of the unfocused or unclear. focused and clear. focused, clear, and Information Some or all of the words Most of the words in complete. All of the Needed in the query are the query are related words in the query unrelated to the to the topic. are related to the (examine information need. topic. question 2)
- 17. Pre-Test to Post-Test Scoring Rubric ACRL Standard Beginning Proficient Advanced 2. Access the Student uses only a Student uses a single Student uses Needed single broad research subject specific multiple subject Information tool (e.g., Google) and research tool and/or specific research Effectively and examines only the first multiple broad tools. Efficiently page of results. research tools. (examine questions 1, 2, and 4)
- 18. Pre-Test to Post-Test Scoring Rubric 3. ACRL Beginning Proficient Advanced Standard Evaluate No mention is made of Student mentions Student mentions Information and criteria that might be one or two of the three or more of the its Sources used to judge following criteria: following criteria: Critically information quality. accuracy, authority, accuracy, authority, (examine credibility, credibility, question 5) relevance, relevance, timeliness. timeliness. On a scale of 1 (very poor) to 9 (very good), how would you rate the student’s information research skills. Base your rating on an overall assessment of their test answers.
- 19. Pre-Test to Post-Test Study Raters: • Four, two scored all pre-tests and two scored all post- tests. •Blind and unaware of study design
- 20. Data Collection
- 21. Pre-Test to Post-Test Improvement Stand 1 pre Stand 1 post Change One-way Anova treatment 1.85 2.43 +.58 F= .124 control 2.0 2.69 +.69 p=0.73
- 22. Pre-Test to Post-Test Improvement Stand 2 pre Stand 2 post Change One-way Anova treatment 1.4 2.05 +.65 F= .649 control 1.3 2.15 +.85 p=0.431
- 23. Pre-Test to Post-Test Improvement Stand 3 pre Stand 3 post Change One-way Anova treatment 2.05 2 -.05 F= 3.032 control 1.85 2.2 +.35 p=0.1
- 24. Pre-Test to Post-Test Improvement rating pre rating post Change One-way Anova treatment 5.45 5.65 +.2 F= .19 control 4.9 5.4 +.5 p=0.668
- 25. Overall Grade related to Posting on AAL Thread Avg. Overall Grade (out of 500) One-way Anova treatment 493 F= 3.35 control 477 p=0.08
- 26. Pre-Test to Post-Test Improvement AAL Posts Read Total Posts Read Std. 1 Aug. 2011 0 -0.4 (p=0.19) Std. 1 Jan. 2012 0.7 (p=0.03) 0.59 (p=0.06) Std. 2 Aug. 2011 0.41 (p=0.12) -0.16 (p=0.33) Std. 2 Jan. 2012 0.24 (p=0.25) 0.11 (p=0.38) Std. 3 Aug. 2011 -0.29 (p=0.21) -0.64 (p=0.02) Std. 3 Jan. 2012 -0.32 (p=0.18) -0.09 (p=0.4) Rating Aug. 2011 0.08 (p=0.41) -0.26 (p=0.23) Rating Jan. 2012 0.46 (p=0.09) 0.17 (p=0.32)
- 27. Overall Grades - correlations AAL Posts Read Total Posts Read Aug. 2011 0.52 (p=0.06) 0.297 (p=0.2) Jan. 2012 0.03 (p=0.47) 0.009 (p=0.49)
- 28. Feedback • I found the requirement to chat with a librarian to be very beneficial in finding new ways to locate information on my location. Overall I enjoyed the class and the variations of assignments that were required of us because they weren't all the same thing which forced us to develop as students. I feel I am leaving this class a much more competent researcher. – (post in reflections on the class thread, Aug. 2011)
- 29. Feedback • Your response does help and I will definitely want to look at Google Scholar and EconLit as they are unfamiliar sources to me. I did feel better about using the data when Tray indicated that the WTTC was cited in this weeks reading. After taking this class and having the opportunity to learn about research sources on the Message Board, I am a much more cautious consumer of information on the internet. Thanks for your response, it is appreciated. – (post on main message board, aug. 2011)
- 30. Feedback • Oh my, this first assignment required some thoughtful research. I assumed it would be multiple choices but it was almost a pleasant surprise. I learned a lot... FUN FUN FUN!! (post in Pre-test thread, Aug. 2011)
- 31. Feedback • I have a question about citing for a video. I found one on YouTube and have really been using the link you gave me on Purdue Owl. Is the "Recorded Films or Movies" section correct for this type of citation? PS I just want to say that both of you guys have really helped me a lot in answering questions and giving valuable and important information for this class. I have used not only the answers you have given me, but answers that you have given my fellow classmates. Thanks to you I may just make it thru this class!(post in AAL thread, Aug. 2011)
- 32. Feedback • I did not know about “Ask a Librarian” feature until I took the library survey for this class. It is really helpful. I utilized this feature two nights now, I love it. (post in AAL Thread, Aug. 2011)
- 34. Student Learning Outcomes Critical Thinking • “Students will demonstrate the ability to access and interpret information” Communication • “Students will demonstrate the ability to communicate clearly and effectively.” Academic & Professional Integrity • “Students will demonstrate awareness and understanding of the ethical standards of their academic discipline and/or profession”
- 35. Summary of Results • Students who posted to a librarian moderated performed better on class grades. • The experience of posting to a librarian moderated thread had no impact on our measure of information literacy skills. • In one class there was a positive relationship between number of AAL posts read and class grades. In the other class there was a positive relationship between the number of AAL posts read and pre to post test improvement on standard 2 and overall rating. • Students improved on our measures of information literacy. We do not know why, exactly. • Some students seemed to enjoy the requirement to do research.
- 36. Next Steps • Be Draconian but helpful • Gather time of exposure as well as number of exposures • Increase incentive to use our services • Improve measures of learning • Perform experiments
Notas do Editor
- Leo has described the context for our study and given some of the details about what we did. I get to do the fun part – describe what happened in the classes and discuss whether it had any impact (good or bad) on the students. Since this conference is using a culinary theme for its tracks, it seems appropriate to say that what we did was attempt to serve information literacy. The main dishes were posts and online support materials. Notice that the bulk of our time was taken up with reading their posts so we knew what they were hungry for and what nourishment they lacked.
- As Leo mentioned, we strove to keep track of the time we spent during the class. Those numbers are in the top row. We estimated our prep time – things like meetings with the instructor, creating Libguides, creating tips documents – in the second row. Overall, the two of us spend somewhere around 25 hours total on these two courses.
- 25 hours of hard work is quite a bit. Was it worth it? One way to approach that question is to examine how much of our meal they consumed. Fortunately, like many LMS systems, ours, KSOL, provides data about the number of posts students made and read and on which threads. Our main interest was in their attention to the Ask A Librarian thread, though we also read the other threads and delivered dollops of information literacy at appropriate times. To be honest, we were disappointed at the low number of students who used the AAL thread to ask questions. But, we were pleasantly surprised to see that many students read all of the posts on that thread (of course they may have merely clicked the links merrily – but we like to ignore that possibility). We did not have a Libguide for the August 2011 intersession. We created one prior to the January 2012 intersession and were pleased to see that it had 407 hits. Unfortunately, we don’t know how substantial the hits were – how long they spent on a page. In August 2011, all students were required to conduct an IM chat and e-mail the transcript to the instructor. That requirement was removed for January 2012 – mostly because Leo and I wanted to have more of the instruction funneled through us. Nonetheless some students still found their way to the chat – a fact that I am happy about.
- Of course we need to do much more than report mere consumption data. With pressures mounting on libraries to prove that they should be funded, it is incumbent upon us to provide evidence that our efforts bring about change. We have taken a multi-faceted approach to providing that evidence. This includes the relationship between our involvement with students and three measures of learning: pre-test to post-test improvement; overall grade; quality of final project bibliographies. It also includes qualitative feedback from the students.
- Sample. Since scoring our pre-test and post-test required effort, we decided to use a sample, rather than the entire population. We didn’t just want any ole sample though. We wanted a sample that would provide some insight into what impact Leo and I had had on their pre-test to post-test improvement, or heaven forbid, decline. We decided to make half of our sample students all students who had posted a question to the Ask A Librarian Thread on the message board and had completed both the pre-test and post-test. Unfortunately, this proved to be only 5 students in each class, or 10 total. Rather than randomly select our control group, we chose to make them equivalent in willingness to post. Therefore we had K-State Online show us a list of students in order by number of posts. We looked for the names of our treatment group (henceforth called askers, er… wise ones) and chose the first person below that name who was not also in the treatment group and who had completed both the pre and post tests. The posting propensity for the students was remarkably similar, so the match turned out to be perfect.
- Leo developed a simple rubric for scoring the pre and post tests. The rubric is based on ACRL’s Information Literacy Standards, 1-3. For each of these standards, we identified which questions on the test provided relevant evidence. We also described attributes of an answer indicative of beginning, proficient, or advanced skill. This is the rubric for ACRL Standard 1 “Determine the Extent of the Information Needed.” We opted to evaluate the students’ ability to do that by assessing the focus of their research queries.
- For standard 2 “Access the needed information effectively and efficiently,” we operationalized that in terms of the nature and number of search tools used to answer 3 of the test questions.
- For standard 3, “evaluate information and its sources critically,” we chose to examine how many criteria they mentioned in their answer to question 5 on the pre and post tests.Finally, we included a rating of overall research skill using a 1 to 9 Likert Scale.
- Raters. I put out a call to my colleagues in UCS asking for volunteers to help score tests. Four colleagues rose to the occasion. We provided 2 of them with all the pre-tests and 2 with all the post-tests, but did not tell them that there were such things as pre and post tests. We also did not tell them that some had used the AAL thread and some had not. Unfortunately, our raters often gave dissimilar scores. This is likely due to a lack of clear directions on our part.
- To gauge the impact of our involvement, we calculated change scores for each student in our sample and then conducted a one-way Analysis of Variance test to see if the average change was different for the treatment and control groups. The results for standard 1 revealed positive change for both groups, but no direct evidence that posting to the board mattered.
- The results for standard 2 are similar.
- And for standard 3, we see the curious result that students who did not post to the AAL board had a nearly statistically significant boost in change score compared to those who did.
- The story with overall rating of research skill is similar to that for standards 1 and 2.
- We also performed a one-way Anova on average grade. Here we found that those students who posted to the AAL thread had higher overall grades than those who did not.
- We were also curious to see if the number of posts read in the AAL thread had any relationship to the pre-test to post-test improvement. The answer is decidedly mixed. In the January 2012 class, there is a positive relationship between the number of AAL thread posts read and improvement on std 1 and the overall rating.
- One of our primary goals for using a message board was to enable other students to read our responses and those of their peers. We hypothesized therefore, that there should be a positive correlation between the number of AAL posts read and measures of learning. This did not prove to be the case with the pre-test to post-test measures, but we did find support for the hypothesis for the August 2011 class.
- With a background in social psychology, I have learned to be skeptical of feedback, especially when it is not given anonymously. Students like to please, and all of us are rather poor judges of causal factors, including assessments of what contributed to their learning. Even so, I was shocked to find that several students expressed appreciation for being required to learn research skills.
- Leo and I are very interested in continuing this line of research. To be able to perform stronger analyses, we need to funnel as much librarian assistance through us as possible. We need to forbid use of general library help services and drive that traffic to the AAL thread. We can make a much stronger impact statement if we can relate amount of time spent interacting with our services to change scores. Mere counts of number of posts read or pages viewed are deceiving because not all views are equal.Education should be about challenging students and providing supports to help them meet those challenges. The instructor we worked with is by her own admission an easy grader. We suspect that students learned that quickly and realized that their minimal efforts would suffice. In the future, we will encourage instructors to raise their expectations of students’ performance – at least on one assignment. The holy grail of assessment involves determining how well students can perform on authentic tasks that matter to them. If we could have them write articles for the student newspaper, post information to Facebook, or make a proposal to student senate, we could find out if they acquired the skills to make compelling arguments. The most compelling evidence will, we believe, come from experiments in which everything is held constant but for the presence or absence of an embedded librarian. Some assert that this may be unethical, but if there is a real threat of librarian support disappearing in the absence of strong evidence, it would be far more unethical not to. And perhaps we could go further and conduct experiments to begin to tease out what types of activities and what level of involvement make a difference. With limited time and the best of intentions, what we need is knowledge about what works and what does not. So, Leo and I ask you to join us and help us all learn to proceed wisely. Together we can all ensure a vibrant future for ourselves and our students.