Enhancing Excellence in Assessment:
Institutional Effectiveness and
Learning Analytics
SUNY Council on Assessment
Learning Analytics Task Group
October 15, 2013

FACT2 and task groups
FACT2:
• “ a well-established venue to foster
collaboration and consensus within a highly
diverse university community.
• “FACT2governance includes representatives
from faculty, librarians, and IT across
individual campuses from all Carnegie
sectors.”
http://fact.suny.edu

FACT2 and task groups
Task groups
• Initiatives developed collaboratively with the
SUNY Provost and the FACT Advisory Council
• 2010-2012 projects:
–
–
–
Learning analytics
E-publishing
Innovative learning space inventory
• 2013 projects:
–
–
–
Learning analytics
Online Accessibility
Experiential Education

2013 Task Group will focus on…
Identifying and sharing best practices
and uses of
Learning Analytics for
assessment
student readiness
course placement
& remedial education

Institutional
Level Use
Advising
Placement
Learning
outcomes
LATG Survey:
course success data is used to
identify disciplines that have lower
than expected success rates;
Data are also correlated with
student characteristics and other
factors like date of registration,
data of application.
Course
outcomes
Instructional
effectiveness
Student
Feedback
Intervention
Support
Persistence
Degree
completion

Institutional Level
a) identify students at risk of leaving the college
without a degree from the college [54%]
b) identify students who are in need of
developmental courses [46%]
c) place students in appropriate credit courses [54%]
d) evaluate student progress on course objectives
[38%]
e) predict student performance in courses [15%]
f) advise students on course selection [54%]
FACT2 Learning Analytics Task Group, CIT May 2013

Interventions & Feedback
FACT2 Learning Analytics Task Group,

Learning Analytics - Working Definition
Learning analytics can be used to…
– diagnose student needs,
– provide feedback to the student, faculty,
instructional developer, and advisor,
– combine with data from other learning
systems to generate new insights about
learning and instruction.

Learning Analytics - Working Definition
• Learning analytics uses software that collects and
analyzes multiple data sets related to the process
of learning to PREDICT and IMPACT student
success.
• This includes data collected in
–
–
–
–
blended and online learning environments,
online portals,
enrollment data,
and other emergent resources connected to the
teaching and learning experience.

New approach
mining the
“pile of big data”
generated by
technology
longstanding approach
“asking research
questions”
and gathering data

Where is the data?
LMS Platform
Analytics
Collected
from explicit student actionscompleting assignments and exams,
From tacit actionsDiscrete
online social interactions,
Analytics
extracurricular activities,
Tools
posts on discussion forums,
and other activities that are not directly
assessed as part of the student’s educational
progress.
FACT2 Learning Analytics Task Group
Stand-alone
Platform
Analytics
Data in online
activities….

Learning Analytics Tools
Approaches
• Institutional
• Faculty and
advising
• Student
assessment and
feedback
Traditional analysis
tools
SPSS

Learning Analytics & Online Learning
Examples of uses…
• Persistence and
retention (APUS)
• Intelligent/adaptive
tutoring (Carnegie
Mellon)
• Research on conditions
that facilitate learning
(CSU Chico)
FACT2 Learning Analytics Task Group

Use Learning analytics to ….
Provide automated feedback to students
Customize course delivery to student learning
styles.
Quizzes or Learning Sequences
in Blackboard or Angel

Use Learning analytics to ….
Provide individualized learning paths to
students based on pre-entry conditions.
Provide adaptive learning paths to students
based on performance in course.
Learning analytics enables tailoring of
responses, such as through adapting
instructional content, intervening with at-risk
students, and providing feedback.

Use Learning analytics to ….
Revise course content, activities,
assessments and/or course structure.

In adaptive learning, the path of each student is highly
personalized

ASSESSMENT AND LEARNING
ANALYTICS

Typically use End of semester
or mid-term data…
Focus on course
outcomes, but no real-time
data for interventions

Learning analytics in
PLACEMENT & COURSE SELECTION

Predictive Analytics: Building Models
Placement for success and
completion..
– which students should be steered
toward which courses? Which
programs?
Can advising process leverage data
on student performance?
– If so, what are the best predictors
of performance?

Predictive Analytics: Building Models
• Can we identify
characteristics of a
successful outcome?
• an unsuccessful
outcome?
“every student with a HS
average of 83 or less, did not
successfully complete the
course…”
DATA SOURCES
Grade in course
Can it be predicted by other
data?
• Major
• High school GPA
• English placement exam
score
• Math placement exam
score
• HS Regent scores….
• SAT Verbal, SAT math
• SAT Writing

PILOTING APPROACHES
FACT2 Learning Analytics Task Group,

Assessment: study habits
Explore the efficacy
of student study
habits

Assessment: study habits
Share information
about what practices
lead to success with
students

Automated feedback with quizzes.
What can we learn from the data?

How does
Where the real
effort lies
Detailed
information
about
thousands
of students
and their
current
status
10/15/2013
2Coach
E
work?
Expertise of hundreds of
students, dozens of instructors
and behavior change experts
MTS
The Michigan Tailoring System: a mature
open-source software system for
creating content designed specifically
for an individual based on data about
that individual
Teaching, Learning, and Analytics at
Michigan
Individually
personalized
messages:
what we all
agree we
would say to
each
student, if
only we
could…

Learning Analytics Opportunities
Leverage…
• institutional practices and tools in place
• interest in using tools for interventions and
student feedback
– More systemic and consistent approach to
placing students in appropriate courses and
developmental courses.
– Especially for online learning
Explore further…
FACT2 Learning Analytics Task Group,

Retention Initiatives

LATG 2013-14 GOALS
1.
Identify and share known best practices and exemplary uses of Learning
Analytics for assessment, and early intervention strategies.
•
2.
Develop and conduct professional development activities for use of learning
analytics for:
•
•
•
3.
Assessment, student feedback, and early intervention activities in a course.
Leveraging existing campus data sources to inform strategies for student readiness, course
placement, and remedial education; and to identify what data is readily available and policy guidance
in the use of data.
Identify ways where learning analytics may help to eliminate some administrative burdens while
improving academic achievement.
Provide opportunities for SUNY faculty to explore Learning Analytics in pilot
projects.
•
•
4.
Identify the common questions for these areas, and share best practices through web resources and
other communication channels.
Develop “Proof of concept” projects through IITG using learning analytics approaches/tools.
Develop a Pilot program that would recruit a small group of faculty and courses to implement
assessment strategies based on analytics, and evaluate. (though Open SUNY initiative?)
Use the finding from best practices research and pilot projects to identify a
course of action for further expansion of Learning Analytics across SUNY.

Adaptive Systems to Pilot or Explore

SUNY Pilot of Learning Analytics Tools
SPSS

QUESTIONS?
Visit SUNY Learning Commons
Learning Analytics Task Group
greg.ketcham@oswego.edu