James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content
•Download as PPT, PDF•
2 likes•2,873 views
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content
Similar to James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content (20)
More from Simon Bates
More from Simon Bates (17)
Recently uploaded
Recently uploaded (20)
James Bond, Monorail Cat and Partying penguins. What happens when you let student design their own assessment content
- 1. James Bond, internet memes and partying penguins (or, what happens when students write their own assessment content) Simon Bates Pearson Strategies for Success Workshop Toronto, May 2013
- 2. 2
- 3. 3
- 6. Overview I. Motivation II. Technology enabler: PeerWise III. Use cases IV. Engagement, learning, question quality?
- 7. Overview I. Motivation II. Technology enabler: PeerWise III. Use cases IV. Engagement, learning, question quality?
- 10. Overview I. Motivation II. Technology enabler: PeerWise III. Use cases IV. Engagement, learning, question quality?
- 11. The University of Edinburgh Edinburgh, Scotland 5th July, 2010 Paul Denny PeerWise bridging the gap between online learning and social media Department of Computer Science The University of Auckland New Zealand
- 12. • Web-based Multiple Choice Question repository built by students • Students: – develop new questions with associated explanations – answer existing questions and rate them for quality and difficulty – take part in discussions – can follow other authors peerwise.cs.auckland.ac.nz
- 14. 14 As a question author…..
- 15. 15
- 16. 16
- 19. 19 As a question answerer …..
- 20. 20
- 21. 21
- 22. 22
- 23. 23
- 24. Overview I. Motivation II. Technology enabler: PeerWise III. Use cases IV. Engagement, learning, question quality?
- 25. Timeline 2010-11: UoE pilot study 2011-12: Multi-institution, multi-course 2012-13: UBC PHYS 101 Coursera MOOC 25
- 26. Pilot year (2010-11) – replace single handin PeerWise was introduced in workshop sessions in Week 5 Students worked through structured example task and devised own Qs in groups. All these resources are available online (see final slide) 26
- 27. An assessment was set for the end of Week 6: Minimum requirements: • Write one question • Answer 5 • Comment on & rate 3 Contributed ~3% to course assessment 27
- 28. 28
- 29. We were deliberately hands off. • No moderation • No corrections • No interventions at all But we did observe….. 29
- 30. • JISC project – SGC4L
- 31. N (students) ~800 N (staff) ~10
- 33. Overview I. Motivation II. Technology enabler: PeerWise III. Use cases IV. Engagement, learning, question quality?
- 34. Generally, students did • Participate beyond minimum requirements • Engage in community learning, correcting errors • Create problems, not exercises • Provide positive feedback 34
- 35. 35
- 36. Generally, students did not • Contribute trivial or irrelevant questions • Obviously plagiarise • Participate much beyond assessment periods • Didn’t all leave it to the last minute 36
- 37. • Phys 101 uptake graph showing midterm
- 39. 39
- 40. 40
- 45. Overview I. Motivation II. Technology enabler: PeerWise III. Use cases IV. Engagement, learning, question quality?
- 46. Correlation with end of course outcomes
- 47. Quartiles Q4 – top 25% Q3 – upper middle Q2 – lower middle Q1 – bottom 25% 22 students did not take the FCI
- 48. 1st year Physics N=172 University of Edinburgh
- 49. 1st year Physics N=172 University of Edinburgh
- 50. Overview I. Motivation II. Technology enabler: PeerWise III. Use cases IV. Engagement, learning, question quality?
- 51. Comprehensive categorisation of >50% of repository for two successive academic years Principal measures to define a ‘high quality question’ - cognitive level of question - explanation quality - other binary criteria
- 52. Category Description 6 Create (synthesise ideas) 5 Assess 4 Analyse (multi-step) 3 Apply (1-step calcs.) 2 Understand 1 Remember Cognitive level of question
- 53. Explanation 0 – Missing 1 – Inadequate (e.g. wrong reasoning / answer, trivial, flippant, unhelpful) 2 – Minimal (e.g. correct answer, but with insufficient explanation or justification, aspects may be unclear) 3 – Good/Detailed (e.g. clear and sufficiently detailed exposition of correct method and answer) 4 – Excellent (e.g. Describes physics thoroughly, remarks on plausibility of answer, use of appropriate diagrams, perhaps explains reasoning for distractors)
- 54. 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 6 Taxonomic Category PercentageofSubmittedQuestions First semesterFirst semester N = 350N = 350 Second semester N = 252Second semester N = 252 Results: Question level Physics 1A / 1B 2011
- 55. Results: Explanation Physics 1A 2010 and 2011
- 56. ‘High quality’ question 1. At least 2/6 on cognitive level (“understand” and above) 2. At least 2/4 on explanation (“minimal” and above) 3. Clearly worded question (binary) 4. Feasible distractors (binary) 5. ‘Most likely’ correct (binary) 6. ‘Not obviously’ plagiarised (binary)
- 57. Results: Physics 1A 2010 and 2011 2 successive years of the same course (N=150, 350) •‘High quality’ questions: 78%, 79% •Over 90% (most likely) correct, and 3/5 of those wrong were identified by students. •69% (2010) and 55% (2011) rated 3 or 4 for explanations •Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level.
- 58. Literature Bottomley & Denny Biochem and Mol Biol Educ. 39(5) 352-361 (2011) •107 Year 2 biochem students •56 / 35 / 9 % of questions in lowest 3 levels. Momsen et al CBE-Life Sci Educ 9, 436-440 (2010) “9,713 assessment items submitted by 50 instructors in the United States reported that 93% of the questions asked on examinations in introductory biology courses were at the lowest two levels of the revised Bloom’s taxonomy”
- 59. • High general standard of engagement and student- generated questions • Relatively few basic knowledge questions • Transferable across disciplines / institutions • We hypothesise scaffolding activities are critical for high level cognitive engagement Summary
- 60. Acknowledgements Physics 101 course team Georg Rieger Firas Moosvi Emily Altiere UBC CWSEI simon.bates@ubc.ca @simonpbates Ross Galloway Judy Hardy Karon McBride Alison Kay Keith Brunton Jonathan Riise Danny Homer Chemistry – Peter Kirsop Biology – Heather McQueen Physics – Morag Casey Comp Sci – Paul Denny
- 61. Community: http://www.PeerWise-Community.org JISC-funded multi institution study:https://www.wiki.ed.ac.uk/display/SGC4L/Home UoE Physics Pilot Study: AIP Conf. Proc. 1413, 359 http://dx.doi.org/10.1063/1.3680069 RSC overview article http://www.rsc.org/Education/EiC/issues/2013January/student-generated-assessment.asp UoE Physics scaffolding resources http://www2.ph.ed.ac.uk/elearning/projects/peerwise/ Resources
- 62. Question quality analysis (1st year Physics University of Edinburgh) Assessing the quality of a student-generated question repository, submitted to Phys Rev, ST Phys Educ Res. Multi-institution, multi-course study Student-generated content: Enhancing learning through sharing multiple-choice questions, submitted to International Journal of Science Education Scaffolding Student Learning via Online Peer Learning, submitted to International Journal of Science Education Publications in preparation / review / press
- 63. Copyright2013GrahamFowell/TheHitman,re-producedwithpermission,EducationInChemistry,Vol50No1(2013)
- 65. Photo credits Photo credits Community: http://www.flickr.com/photos/kubina/471164507/ Screen grab from Mwensch ‘A vision of students today’ http://www.youtube.com/watch?v=dGCJ46vyR9o 65
Editor's Notes
- Socially constructed knowledge and understanding in an online environment.
- There are measures for this – see later
- All (3 minutes) Introductions of each presenter. Who we are and the perspective that we bring.
- All (3 minutes) Introductions of each presenter. Who we are and the perspective that we bring.
- Can follow the author
- All (3 minutes) Introductions of each presenter. Who we are and the perspective that we bring.
- Scaffolding provided in terms of how to come up with material for good questions. The above handout (a blank version was also provided) encourages students to choose topics within their Zone of Proximal Development (Vygotsky), identify misconceptions or things they don’t understand and devise questions with a realistic context. Students got around 45 mins for this activity
- Scaffolding provided in terms of how to come up with material for good questions. The above handout (a blank version was also provided) encourages students to choose topics within their Zone of Proximal Development (Vygotsky), identify misconceptions or things they don’t understand and devise questions with a realistic context. Students got around 45 mins for this activity
- We provided screencasts online to cover details of signing up, logging in, creating questions, finding questions to answer These are now hosted on the PW site. Creating most popular, viewed 170 times (not necessarily all from the 1A course) Assessment set due 9 days later
- We provided screencasts online to cover details of signing up, logging in, creating questions, finding questions to answer These are now hosted on the PW site. Creating most popular, viewed 170 times (not necessarily all from the 1A course) Assessment set due 9 days later
- All (3 minutes) Introductions of each presenter. Who we are and the perspective that we bring.
- We provided screencasts online to cover details of signing up, logging in, creating questions, finding questions to answer These are now hosted on the PW site. Creating most popular, viewed 170 times (not necessarily all from the 1A course) Assessment set due 9 days later
- We provided screencasts online to cover details of signing up, logging in, creating questions, finding questions to answer These are now hosted on the PW site. Creating most popular, viewed 170 times (not necessarily all from the 1A course) Assessment set due 9 days later
- We provided screencasts online to cover details of signing up, logging in, creating questions, finding questions to answer These are now hosted on the PW site. Creating most popular, viewed 170 times (not necessarily all from the 1A course) Assessment set due 9 days later
- Aside from this being a very creative question, with a great diagram, and a good spread of answers it is also a nice example of peer-tutoring The author wrote the question and got the calculation wrong in the first version of the question This was picked up on, and corrected by, another student the same evening, and the version of the question replaced with the peer’s correct explanation We also placed one of the student generated questions on the exam (not this one).
- This is typical of questions submitted by the highest performing students Focussed on quantitative problemsolving * Amusing context * Great care and attention with diags and maths The 4 images show question solutions sketch Solution maths Comment and author’s response.
- All (3 minutes) Introductions of each presenter. Who we are and the perspective that we bring.
- Ross – placeholder for now, to be updated with accurate graphs after 14 th Dec exam
- Force Concept Inventory score is out of 33 Black horizontal bar is the median FCI score for that quartile The blue box shows the middle 50% The whiskers show the highest score and the lowest score in that quartile except for outliers, which are plotted as small circles.
- Ross – placeholder for now, to be updated with accurate graphs after 14 th Dec exam
- A Pearson product-moment correlation was performed which showed that there was a statistically significant positive relationship between the CM score and the end-of-module examination mark (E) in all modules, with a small to moderate correlation rCME , see Table 5. First-order partial correlation was then conducted to explore this relationship, controlling for the pre-module test mark (P). The partial correlation rCME.P was found to be statistically significant, with values between 0.18 and 0.40 (see Table 5). This indicates that there was a positive relationship between the CM score and the end-of-module mark even when taking into account student ability, as measured by their pre-module test mark.
- All (3 minutes) Introductions of each presenter. Who we are and the perspective that we bring.