Gateway Course Success Conference 2015 Preconference Session on Analytics and Student Success

1. USING ANALYTICS TO
IMPROVE STUDENT
SUCCESS:
A PRIMER ON
LEVERAGING DATA TO
ENHANCE STUDENTApril 12, 2015 Matthew D. Pistilli,
PhD

2. Plan for the day
 Introductions and Purpose
 Conceptual Overview
 Other Institutions’ Analytics
 Five Components of Analytics
 Individual/Group Work & Planning
 Managing Expectations in Next Steps
Following along:
 http://bit.ly/JNGIPreCon2015

3. Who are we?
Where are we from?
Why are we here?
Introductions and Purpose

4. Definitions
Student Involvement Theory:
Astin’s Inputs-Environment-Output
Model
Conceptual Overview

5. Definitions

6. Definitions of Learning Analytics
 The measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding and
optimizing learning and the environments in
which it occurs (SoLAR)
 Evaluating large data sets to provide decision
makers with information that can help
determine the best course of action for an
organization, with a specific goal of improving
learning outcomes (EDUCAUSE, 2011)

7. Definitions Continued
 Using analytic techniques to help target
instructional, curricular, and support resources
to support the achievement of specific learning
goals (van Bareneveld, Arnold, & Campbell,
2012)
 the process of developing actionable insights
through problem definition and the application
of statistical models and analysis against
existing and/or simulated future data (Cooper,
2012)

8. Definitions Continued
 Using data to inform decision-making;
leveraging data to identify students in need of
academic support; and allowing direct user
interaction with a tool to engage in some form
of sensemaking that supports a subsequent
action (Krumm, Washington, Lonn, & Teasley)
 The use of data, statistical analysis, and
explanatory and predictive models to gain
insights and act on complex issues (Bichsel,
2012)

9. Data have power!

10. Common Themes

14. Challenge: How do you find the student at risk?
http://www.youthareawesome.com/wp-content/uploads/2010/10/wheres-waldo1.jpg

15. http://www.youthareawesome.com/wp-content/uploads/2010/10/wheres-waldo1.jpg
Challenge: How do you find the student at risk?

16. Key Questions Addressed by Analytics
(Davenport, Harris, & Morison, 2010)
Questions to Be Answered
Past Present Future
Information
What happened?
(Reporting)
What’s happening
now?
(Alerts)
What will
happen?
(Extrapolation)
Insight
How and why
did it happen?
(Modeling,
Experimental
design)
What’s the next
best action?
(Recommendation)
What’s the
best/worst that
can happen?
(Prediction,
Optimization,
Simulation)

17. Analytics is about…
 Actionable intelligence
 Moving research to practice
 Basis for design, pedagogy, self-awareness
 Changing institutional culture
 Understanding the limitations and risks

18. Inputs-Environment-Output
Student Involvement Theory

19. Student Involvement Theory
 Alexander Astin - UCLA
 Involvement:
The amount of physical and psychological
energy that the student devotes to the
academic experience. (1985, p. 134)
 Exists on a continuum, with students investing
varying levels of energy
 Is both quantitative and qualitative
 Direct relationship between student learning and
student involvement
 Effectiveness of policy or practice directly related
to their capacity to increase student learning
(Astin, 1999)

20. Inputs-Environment-Output
Model
Inputs
Outpu
t
Environm
ent

21. Inputs
 The personal, background, and
educational characteristics that
students bring with them to
postsecondary education that can
influence educational outcomes
(Astin, 1984).

22. Inputs
 Astin (1993) identified 146 characteristics, including
 Demographics
 Citizenship
 Ethnicity
 Residency
 Sex
 Socioeconomic status
 High school academic achievement
 Standardized test scores
 GPA
 Grades in specific courses
 Previous experiences & self-perceptions
 Reasons for attending college
 Expectations
 Perceived ability

23. Outcomes
 Basic level
 Academic Achievement
 Retention
 Graduation
 More abstractly
 Skills
 Behaviors
 Knowledge
The things we are
attempting to
develop in
students

24. Environment
 Where we have the most control
 Factors related to students’ experience while
in college
 Astin (1993) identified 192 variables across 8
overarching classifications
Institutional characteristics Financial Aid
Peer group characteristics Major Field
Choice
Faculty characteristics Place of residence
Curriculum Student involvement

25. Data are changing EVERYTHING

26. While “Big Data” Raise
Expectations…
student data drive big decisions.

27. Strategic Intelligence for Higher
Education
How many, how often, where?
Where exactly is the problem?
What actions are needed?
Why is this happening?
What if these trends continue?
What will happen next?
What’s the best that can happen?
Linda Baer,

28. … requires a shift in
thought.
All this data…

29. Moving from…
Data
Describ
es
Decides
to…

30. Other Institutions’ Analytics

31. Austin Peay University
Degree
Compa
ss

32. Rio Salado College
Studen
t
Suppor
t
Model

33. Open Learning Initiative
SNAPP

34. UMBC Purdue
University
Check My Activity

35. Campbell & Pistilli, 2012
Analytics 5 Component Model

36. Modeling retention &
progression
learner
characteristics
instructor
behaviors
fit/learner
perceptions of
belonging
learner
behaviors
course
characteristics
other
supports
retention/
progression
Data Driven
Institutional
Response/Interventions
Data Driven
Institutional
Response/Interventions

37. Five Components of Analytic
Model
Gather
Predic
t
Act
Monito
r
Refine
Compone
nts are
cyclical
starting
with
gather but
can be
drawn
upon at
any point
in the
cycle.

38. Analytic Component 1:
Gather

39. Gather
 Data
 In multiple formats
 From multiple sources
 With insights into students & their success
 That can be analyzed & manipulated into
formulae
Data is the foundation for this work, and
without good data, the effort may be for
naught.

40. Where in the world is your
data?
http://geoawesomeness.com/wp-content/uploads/2013/05/music_map.jpg

41. Gather
 Consider:
 What types of data are collected at your
institution?
 Can you correlate learning data with performance
objectives?
 What is being done with the current data?

42. Gather
 Before gathering, determine what will be
gathered.
 What question are you trying to answer?
 To do so, consider…
 Where will your focus be?
 What data do you already have (or have access
to)?
 What else do you need to collect?
 Who owns that data?
 What will it take to get access to it?
 What are the challenges associated with
assembling all the data?

43. Gather
Ultimately, answer the following questions:
1. How will you describe this analytics area to
interested parties?
2. Who are the key stakeholders that need to be
included in discussions?
3. Who should serve as the lead for this area at
your institution?
4. What other considerations are there?

44. Analytic Component 2:
Predict

45. Predict
Small group discussion
What student behaviors do you think could be used as
predictors of course outcomes?
 Behavior
 Source of data
 Significance

46. Predict
 Begins with the question asked in Gather:
 What do you want to predict?
 How do you identify this as a focus area?
 Prediction models built will be driven by
 Types of data gathered
 Question being answered
 What’s currently being predicted?
 How?
 By whom?
 In what realms? Student success?
 How can you involve those persons in this effort?

47. Predict
 What makes a good model?
 Correlation vs. Causation
 Expertise required
 Data analysis
 Statistical
 Content
 Reliability & Validity
 Frequency of updating
 Challenges & obstacles

48. Predict
 Develop a framework
 Know your student population
 Historical data
 What interventions are feasible
 Rate the predictor

49. Predict
Ultimately, answer the following questions:
1. How will you describe this analytics area to
interested parties?
2. Who are the key stakeholders that need to be
included in discussions?
3. Who should serve as the lead for this area at
your institution?
4. What other considerations are there?

50. Analytic Component 3: Act

51. Act
 Harken back to journalism class…
 Who?
 What?
 Where?
 When?
 Why?
 How?
 Add:
 Available resources?
 Timing

52. Act
 Data-Driven Responses
The value of
an idea lies in
the using of it.
~ Thomas
Edison

53. Act
 The importance of feedback:
The purpose of feedback is to reduce the
gap between current understanding [or]
performance and the desired goal.
(Hattie & Timperley, 2007)

54. Act
 Frequency – more is always better
 Funding the action
 Assessing the impact
 What are you assessing?
 Were behaviors changed?
 How do you know?
 Do different actions need to be:
 Taken (on your end)?
 Suggested (on the students’ end)?

55. Act
Some things to
bear in mind…

56. Act
http://c767204.r4.cf2.rackcdn.com/e5f9bcd4-5617-44df-997b-125563cc3027.jpg

57. Act
< = >

58. Act
62 words
52 words
40 words

59. Act
 A PERFECT
MATCH! Develop more cost
effective student support
services
 Increased efficiency
through more purposeful
student/staff engagement
 Supports a more
personalized approach to
promoting retention and
degree completion

60. Act
Think Pair Share Activity
 Take a few minutes to write a message to a
student performing mediocrely.
 Share your message with a partner – critique
for:
• Length
• Word/Language choice
• Content
 Revise your message based on feedback.

61. Act
Ultimately, answer the following questions:
1. How will you describe this analytics area to
interested parties?
2. Who are the key stakeholders that need to be
included in discussions?
3. Who should serve as the lead for this area at
your institution?
4. What other considerations are there?

62. Act
 Message construction tool
www.purdue.edu/passnote

63. Act – A Final Thought
 Seek to get
The RIGHT information to
The RIGHT people in
The RIGHT way at
The RIGHT time

64. Analytic Component 4:
Monitor

65. Monitor
 Formative & summative in nature
 Can present challenges and obstacles
 It’s a process
 Current process must be understood
 New/parallel processes developed as necessary
 Involving others… to some extent, the more
the merrier
 Availability of resource (time, money, people)
 Timing of monitoring
 Ability to react

66. Monitor
 Review
 Data collected and used… was it
 Necessary?
 Correct?
 Sufficient?
 Predictions made… were they
 Accurate?
 Meaningful?
 Actions taken… were they
 Useful?
 Sustainable?
 Feedback received to date

67. Monitor
Ultimately, answer the following questions:
1. How will you describe this analytics area to
interested parties?
2. Who are the key stakeholders that need to be
included in discussions?
3. Who should serve as the lead for this area at
your institution?
4. What other considerations are there?

68. Analytic Component 5: Refine

69. Refine
 Self-improvement process for
 Analytics at the institution
 The institution
 Enrolled students
 Continual monitoring
 Small tweaks here and there
 Major changes after periods of time
 Updating of algorithms and statistical models
 Outcome data important as
 Assessment
 Additional components for inclusion in the model

70. Refine
 What was learned from this effort?
 Where are the positives?
 Where are the deficiencies?
 Was the goal realized?
 How does the goal/involvement in the project
help meet institutional goals?
 Who else needs to be involved to
improve/enhance the process, actions, and
outcomes?
 How can lessons learned be applied for future
use?

71. Refine
Ultimately, answer the following questions:
1. How will you describe this analytics area to
interested parties?
2. Who are the key stakeholders that need to be
included in discussions?
3. Who should serve as the lead for this area at
your institution?
4. What other considerations are there?

72. Modeling retention &
progression
learner
characteristics
instructor
behaviors
fit/learner
perceptions of
belonging
learner
behaviors
course
characteristics
other
supports
retention/
progression
Data Driven
Institutional
Response/Interventions
Data Driven
Institutional
Response/Interventions

73. Elevator Speech for Project
Determine/solidify Institutional Goal
Work on Component Templates
Activity

74. What is your goal for this project?
What have you learned?
What are your next steps?
What questions do you still have?
Institution Reporting & Town
Hall

75. Managing Expectations in Next
Steps

76. http://i.imgur.com/nZArTnc.jpg

77. Expectations Reality
 Plug and Play
 Immediate results
 Solve every problem
– ever!
 Universal adoption
 Everyone would
love it!
 Fits, starts, reboots
 Mostly long term
outcomes
 Solve some problems,
create some new
problems
 Lackluster use
 Not everyone loved it

78. Institutional Challenges
 Data in many places, “owned” by many
people/organizations
 Different processes, procedures, and
regulations depending on data owner
 Everyone can see potential, but all want
something slightly different
 Sustainability – “can’t you just…”
 Faculty participation is essential
 Staffing is a challenge

79. New Possibilities
 Using data that exists on campus
 Taking advantages of existing programs
 Bringing a “complete picture” beyond
academics
 Focusing on the “Action” in “Actionable
Intelligence”

80. Contact Information
 Email: mdpistilli@jngi.org
 Phone: 317-274-7225
 Twitter: @mdpistilli – twitter.com/mdpistilli

81. References
Astin, A. W. (1984). Student involvement: A developmental theory for higher education. Journal of College Student
Development, 24, 297-308.
Astin, A. W. (1993). What matters in college? Liberal Education, 79(4).
Astin, A. W. (1994). What matters in college: Four critical years revisited. San Francisco: Jossey-Bass.
Bichsel, J. (2012, August). Analytics in higher education: Benefits, barriers, progress, and recommendations
(Research Report). Louisville, CO: EDUCAUSE Center for Applied Research. Available:
http://net.educause.edu/ir/library/pdf/ERS1207/ers1207.pdf
Cooper, A. (2012). What is Analytics? Definition and Essential Characteristics. CETIS Analytics Series, 1(5).
Available: http://publications.cetis.ac.uk/2012/521
Davenport, T. H., Harris, J. G., & Morison, R. (2010). Analytics at work: Smarter decisions, better results.
Cambridge, MA: Harvard Business Press.
EDUCAUSE Learning Initiative. (2011). 7 things you should know about first-generation learning analytics.
Louisville, CO: EDUCAUSE. Available: http://www.educause.edu/library/resources/7-things-youshould-
know-about-first-generation-learning-analytics
Krumm, A. E., Waddington, R. J., Lonn, S., & Teasley, S. D. (n.d.). Increasing academic success in undergraduate
engineering education using learning analytics: A design based research project. Available:
https://ctools.umich.edu/access/content/group/research/papers/aera2012_krumm_learning_analytics.pdf
Oblinger, D. G. and Campbell, J. P. (2007). Academic Analytics, EDUCAUSE White Paper.
Society of Learning Analytics Research. (n.d.) About. [Webpage] Available:
http://www.solaresearch.org/mission/about/