Presented at the Third Learning Analytics and Knowledge conference proceedings at Leuven, Belgium. The presentation talks about an open learning ecosystem and Predictive analytics based Early alert system developed at Marist College. It also researches into how the portable the predictive model can be when deployed in a different academic contexts(community colleges) and gives the results about the model performace.more about OAAI at https://confluence.sakaiproject.org/pages/viewpage.action?pageId=75671025 Conference website: http://lakconference2013.wordpress.com/

1. LAK 2013 – 3rd Conference on Learning Analytics and Knowledge
Open Academic Analytics Initiative:
Initial Research Findings
Eitel J.M. Lauría, Erik W. Moody , Sandeep M. Jayaprakash,
Nagamani Jonnalagadda, Joshua D. Baron
Marist College, Poughkeepsie, NY, USA
Leuven, Belgium April 8 -12 2013

2. Project Overview:
Open Academic Analytics Initiative
OAAI
Eitel J.M. Lauría
School of Computer Science & Mathematics
Marist College

3. Open Academic Analytics Initiative
• Supported by Next Generation Learning
Challenges (NGLC) grant
• Funded by Bill and Melinda Gates and Hewlett
Foundations
• 18 month period, $250,000
• Began mid 2011, we have completed the
project

4. Open Academic Analytics Initiative Objectives
• Create “early alert” system
• Predict “at risk” students in initial weeks of a course
• Deploy intervention to ensure student succeeds
• Based on Open ecosystem for academic analytics
• Sakai Collaboration and Learning Environment
• Pentaho Business Intelligence Suite
• OAAI Predictive Model released under OS license (PMML)
• Collaboration with other vendors (SPSS Modeler)

5. Research Questions
• How good are predictive models ?
• What are good model predictors ?
 Student Attitude Data (SATs, current GPA, etc.)
 Student Demographic Data (Age, gender, etc.)
 Sakai Event Log Data
 Sakai Gradebook
• How “portable” are predictive models?
• What intervention strategies are most effective?

6. Predictive Modeling and
Portability Analysis
Sandeep M. Jayaprakash
Academic Technology & eLearning
Marist College

7. OAAI Early Alert System Overview
SIS or Banner
Student Attitude Data
(Static data)
(SATs, current GPA, etc.)
Student Demographic Data
(Age, gender, etc.)
Identified
Predictive Students at
Model “academic
risk”
(Dynamic Data)
Scoring
SAKAI CLE
Sakai Event Log Data
Sakai Gradebook Data
Intervention
(Awareness & Online Academic
Support Environment)

8. Predictive Modeling using Marist Data
Pentaho Kettle Data Integration
• Training Dataset – Marist Fall 2010 & Spring 2011 (7344 records)
Testing Dataset – Marist Fall 2011 (5101 records )
• Extractions were joined, cleaned, recoded, and powerful predictors were
derived to produce an input data file for each student- course combination.
Feature Type Feature Name
GENDER, SAT_VERBAL,
SAT_M ATH, APTITUDE_SCORE,
FTPT, CLASS, CUM _GPA,
Predictors ENROLLM ENT, ACADEM IC
_STANDING, RM N_SCORE,
R_SESSIONS, R_CONTENT_READ
ACADEM IC_RISK (1 = at risk; 0
Target
student in good standing)

9. Predictive Modeling using Marist Data
Pentaho WEKA 3.7 and IBM SPSS Modeler 14.2
• Generate 10 different training datasets by varying random seeds
• Balance each training dataset using sampling techniques.
• Train a predictive model(Logistic Regression, SVM/SMO, J48
decision Trees) for each balanced training dataset
 10 datasets x 3 algorithms = 30 models
• Score the testing dataset(Marist Fall 2011) for each student-
course combination
• Measure predictive performance of classifiers
 Accuracy, Recall, Specificity and Precision.
• Produce summary measures (mean and standard error)

10. Predictive Modeling using Marist Data

11. Predictive Performance on Marist Data

12. Running Pilots at Partner Institutions
Student Aptitude and AAR transferred from Marist
Demographic Data into a Project Site for faculty at
Extract (SIS) each institutions Sakai system
Pentaho AAR
[data processing,
Project Site
scoring and reporting]
Sakai Event
Academic
Log Data Extract Alert Report
The Sakai
(AAR)
Dropbox tool
is used to
provide each
Gradebook faculty with a Dropbox Tool
Data Extract Open Academic Analytic Initiative private folder
Workflow for Academic Alert
Reports (AAR) and deployment of
Online Academic intervention strategies Faculty Folder
Support Environment
(OASE)
A sub-folder for each course/
section used to organize the Academic Student
AAR and course SIK Alert Report Identification
(AAR) Key (SIK)
Faculty notified when
Messages Tool new AA is posted
Identified
and access their Dropbox
Student
to review AAR
Faculty message
identified
students through the Specific Sakai
Awareness class Course Site Course Site
Intervention

13. Academic Alert Reports (AARs)

14. Predictive Performance on Spring Pilots

15. Portability Analysis
• The models developed at one academic context are scalable to
other academic contexts.
• The evaluation accuracies start at 65 % at the first wave and the
accuracies improves to 75% - 80% with more availability of
data in the subsequent waves.
• Pilot Evaluation results show that recall and specificity
completion values are just around 10% lower when compared
to Marist results.
• Gradebook (CMS data) and CUM_GPA have been very
important predictors.
• Evidence of good portability in institutions collecting such data.

16. Intervention Analysis
Erik Moody
School of Social & Behavioral Sciences
Marist College

17. Intervention Strategies at Partner Institutions
Once “at risk” students had been identified this information could be
used to alert them they are at risk of failing the course.
Last spring three institutions (Cerritos College, College of the
Redwoods and Savannah State University) participated in a pilot study
designed to explore the effectiveness of the predictive model and two
different interventions.
A total of 1,379 students were assigned to one of three groups:
OASE
Control Awareness (Online Academic Support
Environment)
Alerted of Risk of Failure
No Intervention Alerted of Risk of Failure Access to Academic
Support Services

18. Intervention Strategies at Partner Institutions
At three different points during the semester Academic Alerts were
automatically sent to the instructors.
Instructors forwarded the Academic Alerts to the students they felt were
struggling in their course.
Student in the Awareness group were sent emails with messages like:
“Based on your performance on recent graded assignments and exams, as well as
other factors that tend to predict academic success, I am becoming worried about
your ability to successfully complete this class.
I am reaching out to offer some assistance and to encourage you to consider taking
steps to improve your performance. Doing so early in the semester will increase the
likelihood of you successfully completing the class and avoid negatively impacting on
your academic standing.”

19. Intervention Strategies at Partner Institutions
Additionally Instructors were encouraged to recommend, the following:
• Ask the student visit you during office hours.
• Set up an appointment with a tutor, academic support person or consider
participating in a study group.
• Access web-based resources such as online tutoring tools.
• Take practices exams, complete additional & homework questions.
Students in the OASE group received the same messages plus links to
Academic Support Services like The Kahn Academy, Flat World Knowledge
textbooks, etc… as well access to mentoring from peers and professional
support staff.
At the end of the semester we collected data on a number of measures including
course grade, content mastery and course withdrawal.

20. Intervention Analysis (Spring 2012)
Mean Final Grade for "at Risk" Students
100
Final Grade (%)
90
80
70
60
50
Awareness OASE Control
One-way ANOVA analysis revealed statistical significance
differences between the control group and the two
treatment groups. F (2,448) = 8.484, p = .000*

21. Intervention Analysis (Spring 2012)
Content Mastery for "at Risk" Students
500
Frequency 400
300
200
100
0
Yes No Yes No
Control Intervention
X2 analysis reveled a significant difference in content
mastery (C or better) between the control group and the
collapsed treatment groups (X2(1) = 8.913, p = .003*).

22. Intervention Analysis (Spring 2012)
Withdrawal rates for "at Risk" Students
500
400
Frequency
300
200
100
0
Yes No Yes No
Control Intervention
X2 analysis reveled significantly different withdrawal rates
between the control group and the collapsed treatment
groups (X2 (1)=7.097, p = .008*).

23. Intervention Analysis (Spring & Fall 2012)
Mean Final Grade for "at Risk" Students
Final Grade (%) 100
90
80
70
60
50
Awareness OASE Control
One-way ANOVA analysis revealed statistical significance
differences between the control group and the two
treatment groups. F (2, 714) = 7.076, p = .001*

24. Conclusions
Both Treatment groups performed significantly better on
measures of final grade and content mastery than controls.
Both Treatment groups had higher rates of course withdrawal
than controls.
The first of three Academic Alerts were the most effective.
Why do Academic Alerts Help?
• Early feedback is important
• Despite poor grade students may not believe they are at risk
• In large classes students don’t receive the attention they do in
smaller classes

25. Questions

26. Reference
OAAI Sakai confluence Wiki page
https://confluence.sakaiproject.org/pages/viewpage.action?pageId=75671025
Contact
Josh Baron - Senior Academic Technology Officer
josh.baron@Marist.edu
Eitel Lauría - School of Computer Science & Mathematics
eitel.lauria@Marist.edu
Erik Moody - School of Social & Behavioral Sciences
erik.moody@Marist.edu
Sandeep Jayaprakash - Learning Analytics Specialist
sandeep.jayaprakash1@Marist.edu