The document describes an open-source academic early alert and risk assessment application programming interface (API) called the Open Academic Alert Initiative (OAAI). It was developed using big data concepts to create a predictive early alert system for higher education. The system collects student data from learning management systems and other sources, analyzes it using predictive models, and identifies at-risk students. Researchers tested interventions for at-risk students and found they had higher final grades and content mastery compared to control groups. The project aims to develop the technology into an open learning analytics platform consisting of modular components for data collection, storage, analysis and reporting.
SR-101-01012024-EN.docx Federal Constitution of the Swiss Confederation
Open academic early alert & risk assessment ap presentation
1. P r e s e n t e d b y
Members of the Apereo LAI Community
March 18, 2015
Sandeep Jayaprakash, Marist
Gary Gilbert, Unicon
OPEN-SOURCE ACADEMIC EARLY ALERT &
RISK ASSESSMENT API
3. Agenda
Marist early Alert framework
Open Learning Analytics vision
Learning Analytics Processor
Demo
Discussion
4. OAAI: Overview and Impact
EDUCAUSE Next
Generation Learning
Challenges (NGLC)
Funded by Bill and
Melinda Gates Foundations
$250,000 over a 15 month period
Goal: Leverage Big Data concepts to create an
open-source academic early alert system and
research “scaling factors”
5. OAAI: Overview and Impact
Build learning analytics-based early alert system
Sakai Collaboration and Learning Environment
Secure data capture process for extracting LMS data
Pentaho Business Intelligence Suite
Open-source data mining, integration, analysis & reporting
OAAI Predictive Model released under open license
Predictive Modeling Markup Language
Researching learning analytics scaling factors
How “portable” are predictive models?
What intervention strategies are most effective?
6. Student Aptitude Data
(SATs, current GPA, etc.)
Student Demographic
Data (Age, gender, etc.)
Sakai Event Log Data
Sakai Gradebook Data
Predictive
Model
Scoring
Identifies
students
“at risk” to
not
complete
course
SISDataLMSData
OAAI Early Alert System Overview
Intervention Deployed
“Awareness” or Online
Academic Support
Environment (OASE)
“Creating an Open Academic
Early Alert System”
Model Developed
Using Historical Data
Step #1: Developed
model using historical
data
Academic Alert
Report (AAR)
7. Predictors of
Student Risk
Some predictors
were discarded if
not enough data
was available.
LMS predictors were
measured relative
to course averages.
9. Research Design
Deployed OAAI system to 2200 students across four
institutions
Two Community Colleges
Two Historically Black Colleges and Universities
Design > One instructor teaching 3 sections
One section was control, other 2 were treatment groups
Each instructor received an AAR three times during
the semester:
Intervals were 25%, 50% and 75% into the semester
12. Fall ’12 Portability Findings
Conclusion
1. Predictive models
are more “portable”
than anticipated.
2. It is possible to
create generic
models that are
then “tuned” for use
at specific types of
institutions.
13. Intervention Research Findings
Final Course Grades
Analysis showed a
statistically significant
positive impact on final
course grades
No difference between
treatment groups
Saw larger impact in
spring than fall
Similar trend amount
low income students
50
60
70
80
90
100
Awareness OASE Control
FinalGrade(%)
Mean Final Grade for "at Risk" Students
14. Intervention Research Findings
Content Mastery
Student in intervention
groups were statistically
more likely to “master
the content” than those
in controls.
Content Mastery = Grade
of C or better
Similar for low income
students.
0
200
400
600
800
1000
Yes No Yes No
Content Mastery for "at Risk" Students
Control Intervention
Frequency
15. Instructor Feedback
"Not only did this project directly assist my students by guiding
students to resources to help them succeed, but as an instructor,
it changed my pedagogy; I became more vigilant about
reaching out to individual students and providing them with
outlets to master necessary skills.
P.S. I have to say that this semester, I received the highest
volume of unsolicited positive feedback from students, who
reported that they felt I provided them exceptional individual
attention!
16. JAYAPRAKASH, S. M., MOODY, E. W., LAURÍA, E. J.,
REGAN, J. R., & BARON, J. D. (2014). EARLY ALERT OF
ACADEMICALLY AT-RISK STUDENTS: AN OPEN SOURCE
ANALYTICS INITIATIVE. JOURNAL OF LEARNING
ANALYTICS, 1(1), 6-47.
More Research Findings…
17. Strategic Vision: Open Learning
Analytics Platform
Collection
Standards-based data
capture from any
potential source using
Experience API and/or IMS
Caliper/Senor API
Storage
Single repository for all
learning-related data
using Learning Record
Store (LRS) standard.
Analysis
Flexible Learning Analytics
Processor (LAP) that can
handle data mining, data
processing (ETL), predictive
model scoring and
reporting.
Communication
Dashboard technology for
displaying LAP output.
Action
LAP output can be fed
into other systems to
trigger alerts, etc.
18. Technology Stack
Learning Analytics Processor (LAP)
JAVA-based web application
Maven for builds
Temporary Storage - H2 in-memory database
Persistence Storage - MySQL
Predictive Model Mark-up Language (PMML)
OAAI Early Alert Pipeline
Pentaho Kettle – Data Integration & ETL
Pentaho WEKA – Data Mining & Predictive Modelling
21. Features
Key pieces of the LAP architecture
Input source management
Data storage – temporary & persistent
Configuration manager
Pipeline processor
Output results management
Extensibility
Supports multiple pipelines
Supports varied pipeline platforms
22. Demo Overview
● Three core components of a
collection of open source
applications and services that
represent the “Analytics Diamond”
● Can be used individually or
collectively
● Work with a shared infrastructure
and data model
Technologies:
• AngularJS
• Spring-Boot
• Pluggable Datastores
(redis, elasticsearch, mongodb)
OpenLRS
Learning
Analytics
Processor
Sakai
Open
Dashboard
xAPI
LTI
API
API
25. Questions?
APEREO LEARNING ANALYTICS INITIATIVE COMMUNITY
• Accelerate the operationalization of Learning
Analytics software and frameworks
• Support the validation of analytics pilots across
institutions
• Work together so as to avoid duplication
analytics-coordinator@apereo.org
Josh Baron
josh.baron@marist.edu
Sandeep Jayaprakash
sandeep.jayaprakash1@
marist.edu
Gary Gilbert
ggilbert@unicon.net
OK, so what is the OAAI and how are we working to address this problem…with the goal of leveraging Big Data to create an open-source academic early alert system that allows us to predict which students are at risk to not complete the course (and do so early on in the semester) and then deploy an intervention to help that student succeed.
OK, so what is the OAAI and how are we working to address this problem…with the goal of leveraging Big Data to create an open-source academic early alert system that allows us to predict which students are at risk to not complete the course (and do so early on in the semester) and then deploy an intervention to help that student succeed.
OK, so what is the OAAI and how are we working to address this problem…with the goal of leveraging Big Data to create an open-source academic early alert system that allows us to predict which students are at risk to not complete the course (and do so early on in the semester) and then deploy an intervention to help that student succeed.
I’ll talk about our intervention strategies in a little more detail a bit later on in the presentation…
I’ll talk about our intervention strategies in a little more detail a bit later on in the presentation…