Learning Analytics Panel Moderator: Dr. Jill Buban, Senior Director of Research and Innovation, Online Learning Consortium. • https://www.slideshare.net/alexandrapickett/olc-panel-presentation-learning-analytics Panelists: *Patsy Moskal, Associate Director, Research Initiative for Teaching Effectiveness (RITE), University of Central Florida (UCF). *John Vivolo, Director, Online and Virtual Learning for New York University (NYU). *Scott James, Data Scientist, PAR Framework, a division of Hobsons. Open SUNY COTE Summit 2017 - Open SUNY Center for Online Teaching Excellence (https://commons.suny.edu/cotehub/) Annual conference for the SUNY online teaching and learning community of practice. March 8-10, 2017, Syracuse, NY. Conference website: http://opensunycotesummit2017.edublogs.org/ Program: http://opensunycotesummit2017.edublogs.org/about/program/#.WMvhhP21vZs Recordings: http://opensunycotesummit2017.edublogs.org/mediasite/#.WMvkv_21vZs Materials: http://opensunycotesummit2017.edublogs.org/mediasite/#.WMvkv_21vZs Open SUNY COTE: http://commons.suny.edu/cote/

1. Analytics in Higher Education:
A Closer Look at Online Learning
Scott James
Data Scientist, Hobsons

2. Analytics in Higher Ed
Learning Analytics
Learner Analytics
Academic Analytics
Enrollment Analytics
Marketing Analytics

3. Data andAnalytics
Raw Data
Descriptive Statistics (Descriptive Analytics)
Predictive Modeling (Predictive Analytics)
Explanatory Modeling (Prescriptive Analytics)

4. Raw Data

5. Raw Data File
Source: Basketball Reference, http://www.basketball-reference.com/leagues/NBA_2013_advanced.html

6. Raw Data Questions for Higher Ed
• How is data stored at your institution?
• Is academic data mapped to financial data? Student involvement data?

7. • This data indicates a sandy beach on a large body of water, located in a warm climate.
• There appear to be adequate picnic facilities for 8 groups.
• Given the low angle of the sun, it is likely the data was collected in the late evening or early morning.
Descriptive
Analytics

8. Descriptive Questions for Higher Education
• How have second year retention rates changed over the past five years?
• How many of our current students are adult learners?
• What was last year’s retention rate for adult minority students?
• What percent of fully online students are part-time?

9. Notes on Example Data
• Contains a random sample of 15,000 students from a four year university who first enrolled between 2009 and
2014
– 10,000 past students to explore past data and build a model, and 5,000 ‘current’ students to build predictions for
• Retention rates and predictions are based on retention to a second year
• Credits attempted and part-time/full time status are taken from a student’s first fall or spring term
• Fully Online: Students who took 100% of their course attempts in an online setting
• Fully Onground: Students who took 100% of their course attempts on-campus
• All Others: Students who took both online and onground courses, or courses in other delivery modes such as
hybrid courses

10. Number of Students by Delivery Mode

11. Percent of Students who are 25 and Older by Delivery Mode

12. Percent of Students who are Part-Time by Delivery Mode

13. SecondYear Retention by Delivery Mode

14. • 80% chance that sunset occurred within an hour of this data collection point.
• 85% chance that none of the grills are in use.
• 90% chance of sunburn if the data collector is Scott.
Predictive
Analytics

15. Prediction Questions for Higher Education
• How likely is a student to retain to the next year?
• How likely is a student to pass College Algebra?
• How likely is a student to graduate in four years?

16. Predicted Retention by Delivery Mode

17. Predicted Retention by Delivery Mode
• Target students
whose probability of
retention is
between .40 and .
60 for new
intervention
• Note that many of
these students will
be online learners

18. PredictiveAnalytics Recap
• We used predictive analytics to score students by their probability of second year retention
• Used the probabilities to target students for new interventions
• Identified that many of the targeted students are online learners
• Will likely want to tailor at least one intervention to meet the specific needs of online learners
• Retention probabilities and individual student characteristics can be great information for advisors

19. Beware of the Jump to Causation
Causation

20. Jumping to Causation
Take action by:
• Nudging online learners to take courses in person?
• Eliminating online degree offerings?
We have identified online students are less likely to be retained. We have not identified a causal
mechanism that explains this

21. • When you visit this location, wear casual attire appropriate for the beach in a warm climate.
• Bring your own picnic supplies and food if you plan to eat.
• Scott should apply sunscreen liberally.
Prescriptive
Analytics

22. Prescriptive Questions for Higher Education
• How effective is the current tutoring program for entry math courses?
• What students benefit the most from student success courses?
• How does online course taking impact a student’s chances of retention once other factors are
accounted for?

23. Predicted Retention by CreditsAttempted

24. Predicted Retention by CreditsAttempted and Delivery Mode

25. Predicted Retention by CreditsAttempted
12 or more credits
attempted
Less than 12 credits
attempted

26. Predicted Retention for Part-Time Students Only
A majority of part time
students are fully
online
The distribution of
retention probabilities
is similar for fully
onground and fully
online students after
isolating part-time

27. Predicted Retention for Full-Time Students Only
Very few fully online
students are full time,
making it difficult to
compare the
distributions

28. Research Question
Are online learners at risk of stopping out due to online
course taking itself, or due to other factors that differentiate
online learners from on-campus learners (i.e. Total Credit
Attempts, Student Age, Financial Need, etc.)?

29. Research Findings
Variables Controlled For:
• GED
• Gender
• Pell Grant
• Race
• Student Age
• Transfer
• Undeclared Major
• Developmental Education
Course Attempts
• Prior Credits
• Course Credits Attempted

30. Research Findings
Fully Online students had 1.24 times greater odds of stopping out than Fully
Onground students. This is a small effect.
Effect of Delivery Mode Estimate 95%
Confidence
Limits
Statistically
Different
from 1.0?
Effect Size
All Others vs Fully
Ongound
1.12 (.98 , 1.3) No None
Fully Online vs Fully
Onground
1.24 (1.07, 1.42) Yes Small

31. Interpretation andAction
Final Take-Aways
• Fully online students have lower probabilities of second year retention according to the model
• Differences in predicted retention between fully online and fully onground students are mostly
explained by the amount of credits students are attempting and demographic factors

32. Interpretation andAction
Action Items
• Meet with fully online students to better understand their unique needs
• Work to make degree plans more flexible and relevant to the needs of fully online learners
• Look for ways to make fully online students feel more connected to the university

33. Questions?