Are Great Wikis Born or Made? Are Students Just Posting in the Same Place?
•Transferir como PPTX, PDF•
1 gostou•1,368 visualizações
A pair of papers for AERA #2012 derived from research from the DCLC project (edtechresearcher.org).
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (11)
Semelhante a Are Great Wikis Born or Made? Are Students Just Posting in the Same Place?
Semelhante a Are Great Wikis Born or Made? Are Students Just Posting in the Same Place? (20)
Mais de EdTechTeacher.org
Mais de EdTechTeacher.org (20)
Último
Último (20)
Are Great Wikis Born or Made? Are Students Just Posting in the Same Place?
- 1. Are Great Wikis Born or Made? Evaluating Quality Trajectories in Wiki Learning Environments Justin Reich Richard Murnane John Willett Just Posting in the Same Place? Evaluating Student Collaboration in Wiki Learning Environments @bjfr edtechresearcher.org
- 2. Distributed Collaborative Learning Communities Project: Web 2.0 in K-12 Settings • Excellence: How do we make them good? • Equity: Do only certain kids get the good ones? • Analytics: What can we learn about learning from real-time usage data from online learning environments 2
- 3. 3 Initial Quantitative Analysis to Develop Sampling Strategy Classroom Observations and Teacher Interviews to Understand Wiki Practices Measure Wiki Quality Develop Wiki Quality Trajectories Literature Review of CSCL and 21st C. Skill Scholarship Path Diagram of Wiki Research Correlate wiki quality with school level SES
- 4. 4 Initial Quantitative Analysis to Develop Sampling Strategy Classroom Observations and Teacher Interviews to Understand Wiki Practices Measure Wiki Quality Develop Wiki Quality Trajectories Literature Review of CSCL and 21st C. Skill Scholarship Path Diagram of Wiki Research Correlate wiki quality with school level SES
- 5. Low Income Schools (n=117) Mid to High Income Schools (n=133) Failed or Trial Wiki 50% 30% Teacher-Content Wiki 34% 35% Individual Student-Owned Wiki 15% 35% Collaborative Student-Owned Wiki 2% 1% Median Lifetime 6 days 33 days
- 6. 6 Initial Quantitative Analysis to Develop Sampling Strategy Classroom Observations and Teacher Interviews to Understand Wiki Practices Measure Wiki Quality Develop Wiki Quality Trajectories Literature Review of CSCL and 21st C. Skill Scholarship Path Diagram of Wiki Research Correlate wiki quality with school level SES Are great wikis born or made?
- 7. “Time is Precious” • Reimann (2009) – Collaborative learning unfolds over time • Substantive need – Online learning environments track data in real time • Methodological opportunity
- 8. 8 Initial Quantitative Analysis to Develop Sampling Strategy Classroom Observations and Teacher Interviews to Understand Wiki Practices Measure Wiki Quality Develop Wiki Quality Trajectories Literature Review of CSCL and 21st C. Skill Scholarship Path Diagram of Wiki Research Correlate wiki quality with school level SES How did we measure collaborative behaviors on wikis, and what did we find?
- 9. Theoretical debates in wikis and collaboration (IJCSCL) Are students meaningfully collaborating? Dohn (2009): Web 2.0 tools are theoretically well-suited for collaborative knowledge production practices, but the educational logics of individual assessment prevent these practices from being realized. Or just posting in the same place? Glassman and Kang (2011): wikis not only enable students to participate in knowledge building, but allow the enactment of Dewey’s vision
- 10. The Plan • Explain our research design • Explain how we model wiki quality over time – Answer the question: Are Great Wikis Born or Made? • Explain in detail how we measure collaboration – Answer the question: Who’s right: Dohn or Glassman & Kang?
- 11. RESEARCH DESIGN
- 12. Which wikis are in my dataset? • All 179,851 publicly-viewable education-related wikis started on the PBworks platform between June 2005 and August of 2008. • Does not include “private” wikis (~70,000) 12 179,851 PBWorks Wikis
- 13. Which wikis are in my sample? • Randomly sampled 1,799 wikis (1%) • Coded to identify 406 U.S. based, K-12 wikis • 255 from specific, identifiable public schools – Detailed usage statistics provided by PBworks.com – School level SES data from the Common Core of Data (National Center for Education Statistics, 2007-2008) – 255 wikis are from 41 U.S. states 13 179,851 PBWorks Wikis 1,799 1% Random Sample 406 U.S. K-12 Wikis 255 Public School
- 14. How should we measure wiki quality?
- 15. Wiki quality as opportunities for 21st Century Skill Development Expert Thinking Complex Communi- cation New Media Literacy
- 16. Wiki Quality Instrument • Information Consumption (2 items): – Do students use wikis to get information? links? • Participation (4 items): – Do they contribute? Do they own pages? • Expert thinking (5 items): – Do students use academic content knowledge in wiki activities? – Do students reflect on the process/product? • Complex Communication/Collaboration (7 items): – Do students concatenate text on pages? – Do they substantively edit each others work and co-create pages? • New Media Literacy (6 items): – Do students use formatting? – Do they hyperlink? Embed media? 24 Questions Wiki Quality Scale of 0-24
- 17. How did we measure wiki quality? • Sample wiki quality at 7, 14, 30, 60, 100, and 400 days • Two raters independently apply wiki quality instrument – All raters must code “training set” of wikis within 1.5 points of master coders – Weekly meetings while coding to discuss categories, difficult cases, etc. • Third rater reconciles disagreements 17
- 18. Data Analytic Strategy: Multilevel Model for Change • We should have time nested within wikis and use the MMFC to model quality trajectories • What functional form should we use to represent time? • The decision that we make in setting a functional form will answer the research question about time!
- 19. Data Analytic Strategy: Multilevel Model for Change
- 20. Selected Empirical Growth Plots
- 21. Empirical Velocity Plot of Wiki Quality 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 100 200 300 400 500 Mean Difference in Score/Days
- 22. Data Analytic Strategy: Multilevel Model for Change
- 23. Comparing Fit Statistics Linear Quadratic Log Intercept 2.8371*** 2.5647*** 1.2073*** Time 0.005629*** 0.02078*** Time2 -0.00004*** Ln(Time) 0.6614*** -2LL 5542.2 5468.7 5263.0 AIC 5554.2 5482.7 5275.0 BIC 5578.3 5510.9 5299.1
- 24. Data Analytic Strategy: Multi-level Poisson Regression Singer and Willett (2008); Rabe-Hesketh and Skrondal (2008)
- 25. Prototypical wiki quality trajectory (n=406) 0 5 10 15 20 0 100 200 300 400 Composite Wiki Quality Score Days
- 26. Implications • Early wiki quality is a strong predictor of later wiki quality- Great Wikis are Born – Hypothesis: Norms developed early on in wiki learning communities are powerful in shaping later use – FOR TEACHERS: Therefore, wiki design deserves critical attention – FOR RESEARCHERS: Further qualitative studies of wiki using communities should ensure that observers can study wikis from the start of the process
- 28. Theoretical debates in wikis and collaboration (IJCSCL) Are students meaningfully collaborating? Dohn (2009): Web 2.0 tools are theoretically well-suited for collaborative knowledge production practices, but the educational logics of individual assessment prevent these practices from being realized. Or just posting in the same place? Glassman and Kang (2011): wikis not only enable students to participate in knowledge building, but allow the enactment of Dewey’s vision
- 29. 29 Initial Quantitative Analysis to Develop Sampling Strategy Classroom Observations and Teacher Interviews to Understand Wiki Practices Measure Wiki Quality Literature Review of CSCL and 21st C. Skill Scholarship Path Diagram of Wiki Research
- 30. Complex Communication Concatenation Do multiple students add discrete sections of text to the same page? Copyediting Does at least one student copyedit text created by another student? Co-ConstructionDoes at least one student substantively edit text created by another student? Commenting Does at least one student comment upon another student’s work on the wiki? Discussion Do students respond to each others’ comments for at least four conversational turns? Scheduling Do students schedule meetings or tasks? Planning Do students plan for future work?
- 31. Concatenation Concatenation Do multiple students add discrete sections of text to the same page? Copyediting Does at least one student copyedit text created by another student? Co-ConstructionDoes at least one student substantively edit text created by another student? Commenting Does at least one student comment upon another student’s work on the wiki? Discussion Do students respond to each others’ comments for at least four conversational turns? Scheduling Do students schedule meetings or tasks? Planning Do students plan for future work?
- 32. Complex Communication Concatenation Do multiple students add discrete sections of text to the same page? Copyediting Does at least one student copyedit text created by another student? Co-ConstructionDoes at least one student substantively edit text created by another student? Commenting Does at least one student comment upon another student’s work on the wiki? Discussion Do students respond to each others’ comments for at least four conversational turns? Scheduling Do students schedule meetings or tasks? Planning Do students plan for future work?
- 33. Co-Construction Concatenation Do multiple students add discrete sections of text to the same page? Copyediting Does at least one student copyedit text created by another student? Co-ConstructionDoes at least one student substantively edit text created by another student? Commenting Does at least one student comment upon another student’s work on the wiki? Discussion Do students respond to each others’ comments for at least four conversational turns? Scheduling Do students schedule meetings or tasks? Planning Do students plan for future work?
- 34. Complex Communication Concatenation Do multiple students add discrete sections of text to the same page? Copyediting Does at least one student copyedit text created by another student? Co-ConstructionDoes at least one student substantively edit text created by another student? Commenting Does at least one student comment upon another student’s work on the wiki? Discussion Do students respond to each others’ comments for at least four conversational turns? Scheduling Do students schedule meetings or tasks? Planning Do students plan for future work?
- 35. 0 5 10 15 20 0 50 100 150 200 250 300 350 400 Complex Communication Score Days Prototypical wiki trajectory for Complex Communication WQI subdomain (n=406)
- 36. Selected Empirical Growth Plots Rate Day Composite Wiki Quality Score
- 37. Distribution of Complex Communication subdomain score in 406 U.S., K-12 wikis on final occasion of measurement Complex Communication Score Percentage Frequency Count 0 88.92 361 1 6.40 26 2 1.48 6 3 .25 1 4 1.48 6 5 .74 3 6 0 0 7 .74 3
- 38. Distribution of collaborative behaviors in 406 U.S., K-12 wikis on final occasion of measurement. Percentage Frequency Count Concatenation 5.91 24 Copyedit 1.72 7 Co-construction 4.43 18 Comment 6.4 26 Discussion 2.22 9 Planning 2.46 10 Scheduling 1.72 7
- 39. 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Information Consumption Participation Expert Thinking NewMedia Literacy Complex Communication Wiki Quality Subdomain Scores (Averages for each WQI Level) (n=255)
- 40. SourceForge • 70% of projects have only 1 developer, • 14% of projects have 2 developers • 14% of projects have 3-9 developers • 2% of projects have more than 10 developers
- 41. Theoretical debates in wikis and collaboration (IJCSCL) Are students meaningfully collaborating? Dohn (2009): Web 2.0 tools are theoretically well-suited for collaborative knowledge production practices, but the educational logics of individual assessment prevent these practices from being realized. Or just posting in the same place? Glassman and Kang (2011): wikis not only enable students to participate in knowledge building, but allow the enactment of Dewey’s vision
- 42. Implications • Empirical: Few wikis provide opportunities for student collaborative writing – Very little copyediting or rich discussion – “I DON’T CARE DO UR OWN PAGE!,” (Grant, 2009).
- 43. Implications • Hopeful: Research to develop strategies for scaffolding online student collaboration is really important • Despairing: What does it mean if Dohn is right? What if the institutional structures of schooling limit collaborative learning?
- 47. Back Deck
- 48. What subjects are wikis used for? (n=411) 2 4 5 6 8 10 20 22 26 45 60 61 70 120 0 20 40 60 80 100 120 140 Education Classics ESL Business Health/PE Modern FL Contained Elementary Art Library Math Computer Science/ Technology Science Social Studies English / Language Arts
- 49. 49 1,000K 10K 1,000 100 10 1 100K # of Cases Days Weeks Months Years Design Research Interviews Surveys Simulations Semantic Analysis Content Analysis Discursive Analysis Observational Research Duration of data collection and capture Time/Scale Web 2.0 Research State Space Modeling Usage Statistics Seconds Biometric Analysis
- 50. Prototypical wiki quality trajectories in subject areas, controlling for SES (n=259) 0 5 10 15 20 25 0 100 200 300 400 500 Wiki Quality Score Days Social Studies English Computer Science Science Math
- 54. 0 10 20 30 40 50 60 70 80 90 100 0 50 100 150 200 250 300 Days MrBoyersClass.Pbworks.com Page Saves by Day Page Saves
- 55. AL L PS 0 1 0 2 0 3 0 d a y 0 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 6 0 0 7 0 0 8 0 0 9 0 0 Moving average of wiki development measured in page saves
Notas do Editor
- We took all 179,851 publicly viewable education related wikis created on the Pbworks platform from the start of the company in June 2005 through August of 2008.
- We randomly sampled from these wikis to get a 1% sample of 1,799, and then we separated out all of the wikis used in other countries, or in higher education, or that were unidentifiable and we ended up with 411 wikis used in U.S. K-12 settings. We then focused in on a subsample of 255 wikis used in specific, identifiable public schools. What’s exciting about this random sample is that we have both detail usage statistics from Pbworks, we can example the entire historical content of each wiki, and since these wikis are linked to specific public schools we can obtain school level SES data from the Common Core of Data. We were very happy to see that these 255 wikis come from 41 US states, which provides strong evidence that our random sample can be used to characterize the state of wiki usage across the U.S.
- The wiki quality instrument is a coding instrument used by trained researcher, and it consists of 24 questions about different kinds of behaviors typically encountered on wikis in five different categories. REVIEW BRIEFLY. So you can imagine that our research assts sit down in front of a computer, they pull up a special browser interface that allows them to easily navigate the historical record of wikis, and the look at the revision history and evaluate which of these 24 behaviors are present. The items ask questions LIKE. So from these 24 dichotomous questions, we can assemble a Composite Wiki Quality Score that ranges from 0 to 24.
- So from these empirical analyses and from a comprehensive modeling strategy we ended up selecting a multi-level Poisson regression model as our final model. And I want to put this model up here to signal methodological competence, to let you know that we did some pretty sophisticated, thorough, careful modeling, but actually I’m not going to take the time unpack this, and at the end of it all we convinced ourselves that you could very responsibly summarize these wiki quality trajectories not with complicated models by simply reporting the wiki quality score on day 14
- Let’s go back to these plots, and pull up one example Essentially, since wiki quality growth is logarithmic, rapid early on, leveling afterwards, so for most purposes rather than using a complex multilevel statistical machinery to represent the whole wiki quality trajectory, for many purposes you can use just use wiki quality scores at day 14 as a single number, comparable across wikis that persist for different lengths of time, that summarizes wiki quality trajectories. It’s a little bit like using median lifetime as a summary statistic for a survivor function. So we’ve tried to find the simplicity on the far side of complexity.