1. Learner Analytics
Realizing their Promise in the CSU
John Whitmer, CSU Office of the Chancellor & CSU Chico
Kate Berggren, CSU Northridge
Hillary Kaplowitz, CSU Northridge
Tom Norman, CSU DH Download slides at:
http://bit.ly/HqaHBF

2. Outline
1. Promise of Learner Analytics
2. Tools & Systems in Practice
3. CSU Case Studies:
• Analytics at Work in the Classroom (Hillary)
• GISMO & SQL Query Tools (Kate)
• Vista in RELS 180 (John)
4. Q & A

3. 1. PROMISE OF LEARNER ANALYTICS

4. Steve Lohr, NY Times, August 5, 2009

5. Draft DOE Report
released April 12
http://1.usa.gov/GDFpnI

6. Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire
economy. The Economist.

7. Source: jisc_infonet @ Flickr.com
7
Source: jisc_infonet @ Flickr.com

8. What’s different with Big Data?
4 V’s:
1. Volume
2. Variety
3. Velocity
4. Variability
(IBM & Brian Hopkins, Forrester)
8

9. Academic Analytics
“Academic Analytics marries large data sets with
statistical techniques and predictive modeling to
improve decision making”
(Campbell and Oblinger 2007, p. 3)

10. Academic Analytics
1. Term adopted in 2005 ELI research
report (Goldstein & Katz, 2005)
– Response to widespread adoption ERP
systems, desire to use data collected
for improved decision making
– 380 respondents; 65% planned to
increase capacity in near future
2. Call to move from
transactional/operational
reporting to what-if analysis,
predictive modeling, and alerts
3. LMS identified as potential domain
for future growth 10

11. DD Screenshot

12. Learner Analytics:
“ ... measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding and
optimizing learning and the environments in
which it occurs.” (Siemens, 2011)

13. or said plainly:
 What are students doing?
 Does it matter?

14. Learner Analytics
1. Analyze combinations of data including:
– Frequency of ed tech usage (e.g. clickstream analysis)
– Student learning “outputs” (e.g. quiz scores, text answers)
– Student background characteristics (e.g. race/ethnicity)
– Academic achievement (e.g. grades, retention, graduation)
2. Current rsch: mostly data mining, not hypothesis-driven
3. More complex than Academic Analytics, considering:
– Immaturity of ed tech reporting functionality
– Translation of usage into meaningful activity
– No significant difference: not what technology used, it’s how
it’s used, who uses it, and for what purpose

15. A few promises of analytics for faculty
and students …
1. Provide behavioral data to investigate student
performance
2. Inform faculty about students succeeding or at
risk of failing a course
3. Warn students that they are likely to fail a
course – before it’s too late
4. Help faculty evaluate the effectiveness of
practices and course designs
5. Customize content and learning activities
(e.g. adaptive learning materials)

16. What’s the promise of analytics for
academic technologists?
1. Decision-making based on actual practices (not
just perceptions) and student outcomes
2. Support movement of A.T. into strategic role re:
teaching and learning by:
– demonstrating the link between technology
and learning
– distinguishing our role from a technology
infrastructure provider

17. Our 2 biggest barriers
Image Source: http://bit.ly/Hq9Cdg

18. Image Source: Utopian Inc http://bit.ly/Hq9sCq

19. Image Source: Privacy in the Cloud: http://bit.ly/HrF6zk

20. 2. TOOLS & SYSTEMS IN PRACTICE

21. SIGNALS
Purdue Signals Project http://www.itap.purdue.edu/studio/signals/

22. SNAPP
SNAPP (Social Networks Adapting Pedagogical Practice) http://www.snappvis.org/

23. KHAN
Khan Academy http://www.khanacademy.org/

24. OLI
CM Open Learning Initiative http://oli.web.cmu.edu/openlearning/initiative/process

25. PARCHMENT
Parchment http://www.parchment.com/c/my-chances/

26. 3. CSU CASE STUDIES

27. ANALYTICS AT WORK IN THE
CLASSROOM (HILLARY)
27

28. How can data help teachers
and students?
Two stories about how data helped students
and teachers work better together
28

29. 29

30. 30

31. “Hey Professor,
I just looked at my assignments and
realized that my Chapter 11 summary did
not get submitted, which I'm having
trouble believing that I didn't submit it...
especially because I see that I did it, and I
always submit my assignments as soon as I
finish them.”
31

32. Now the hard part….
Do I believe him?
If I only I could check…
32

33. 33

34. 34

35. And it was all his idea…
The student suggested that I check Moodle and
if that didn’t work told me how to check the
Revision History in GoogleDocs with step-by-
step directions!
35

36. 36

37. Hybrid Course Weekly Structure
4. Post
3. Online questions
1. Watch 2. Read 4. Class 5. Aplia
chat and and take
lectures textbook meets quiz
tutoring practice
quiz
37

38. “The quiz is unfair”
38

39. But the story was not that simple…
» Reports on Moodle painted a different picture
» Student was watching the lectures at 10:00 p.m.
» Then immediately taking quiz
39

40. Enabled constructive feedback…
1. Advised the student how the structure of the
course was designed to enhance learning
2. Student revised their study habits
3. Improved grades and thanked the instructor!
40

41. What we can do with data now
1. Use Reports in Moodle to verify student claims
2. Review participant list to see last access time
3. Empower students to review their own reports
4. Analyze usage and advise students how to study better
5. Review quiz results to find common misconceptions
41

42. And if we had better tools
that are easier to use…
1. Let our students see more details about how their habits
affect their grades and encourage them to use them
2. Give instructors access to more information and better tools
to organize data so they can see patterns of access and time
on task and how they relate to outcomes
3. Have tools that red flag students with teacher set criteria
4. Help streamline workflow for instructors by organizing
student information
– View all ungraded assignments
42

43. Could we help improve student
learning outcomes if we knew the
effect of…
Coffee
Friends Time
Attendance Amount
Mobile Textbook
LMS
LMS Access
Activities 43

44. GISMO & SQL QUERY TOOL (KATE)
44

45. GISMO – Course Block
45

46. GISMO – Access Overview
46

47. GISMO – Access by Student
47

48. GISMO – Quiz Overview
48

49. SQL Query Tool
49

50. List of Contributed Queries
50

51. Query Example
51

52. Query Results:
Most Active Courses
52

53. Query: Most Popular Activities
53

54. Query:
Systemwide use of
Activities and Resources
54

55. Query:
Forum Use Count by Type
55

56. Learner Analytics
Thomas J. Norman
California State University
Dominguez Hills

57. eBook

58. A New Way of Reading

59. From Textbooks to Apps

63. Assignments

64. Grading To Do List

65. Real Time Metrics

66. Warnings

67. LearnSmart Progress

68. Analysis by AACSB Categories

69. Bloom’s Taxonomy

70. Performance by Learning
Objective/Difficulty

71. Ideas? Questions?
• tnorman@csudh.edu
• tom@professornorman.com
• 310-243-2146

72. CSU CHICO VISTA ANALYTICS
72

73. LMS Learner Analytics @ Chico State
Campus-wide
– How are faculty & students using the LMS?
– What meaningful activities are being conducted?
– How does that usage vary by student background, by college, by
department?
Course level
– What is the relationship between LMS actions, student
background characteristics and student academic achievement?
(6 million dollar question)
– Intro to Religious Studies: redesigned in Academy eLearning,
increased enrollment from 80 to 327 students first semester
Ultimate goal: provide faculty and administrators with what-if
modeling tools to identify promising practices and early alerts
73

74. 74
Chart from Scott Kodai, Chico State

75. CSU Practice

76. INTRODUCTION TO RELIGIOUS
STUDIES (RELS 180)

80. CLOSING THOUGHTS
80

81. Call to Action
1. Metrics reporting is the foundation for Analytics
2. Don’t need to wait for student performance
data; good metrics can inspire access to
performance data
3. You’re *not* behind the curve, this is a rapidly
emerging area that we can (should) lead ...
4. If there’s any ed tech software folks in the
audience, please help us with better reporting!

82. Want more? Resources on Analytics
 Googledoc: http://bit.ly/HrG6Dm

83. Q&A and Contact Info
Resources Googledoc: http://bit.ly/HrG6Dm
Contact Info:
• John Whitmer (jwhitmer@csuchico.edu)
• Hillary C Kaplowitz (hillary.kaplowitz@csun.edu)
• Berggren, Kate E (kate.berggren@csun.edu)
Download presentation at:
http://bit.ly/HqaHBF
83