1.
Machine learning algorithms as tools for
student success prediction
Assoc. Prof. Dijana Oreški, PhD
University of Zagreb
Faculty of Organization and Informatics

2.
Intro
• This work has been supported by Croatian Science
Foundation under the project UIP-2020-02-6312.
 SIMON – Intelligent system for automatic selection of machine
learning algorithm in social sciences
• Lab Louise
 Laboratory for data mining and intelligent systems
 louise.foi.hr

3.
Intro
• “If you torture data long enough they will confess at
the end.”
Ronald Coase

4.
Agenda
• Introduction
 Education data & WHY?
 machine learning algorithms
• Research motivation
• Research methodology HOW?
 CRISP DM standard
• Research results WHAT?
 Model evaluation
 Model interpretation
• Conclusion

5.
Agenda
• Introduction
 Education data & WHY?
 machine learning algorithms
• Research motivation
• Research methodology HOW?
 CRISP DM standard
• Research results WHAT?
 Model evaluation
 Model interpretation
• Conclusion

6.
Introduction
• Huge number of machine learning algorithms
applications in a broad spectrum of domains.
 Crucial role in harnessing the power of the vast amount of data
we produce daily in the digital age.
• The application of algorithms is complex, iterative and
time-consuming.
 There is a need to automate the selection of algorithms for
models development.
• Which algorithm is best to used in a specific situation,
in a particular domain, at a particular dataset?

7.
Research motivation

8.
Research motivation
Education
data
Supervised
machine
learning
algorithms

9.
Agenda
• Introduction
 Education data & WHY?
 machine learning algorithms
• Research motivation
• Research methodology HOW?
 CRISP DM standard
• Research results WHAT?
 Model evaluation
 Model interpretation
• Conclusion

10.
CRISP DM standard
Data
preparation
Data
understanding
Evaluation
Modelling
Data
Domain
understanding
Deployment

11.
Research papers
• Kliček, B.; Oreški, D.; Divjak, B., Determining individual learning
strategies for students in higher education using neural networks,
International Journal of Arts and Sciences.
• Oreški, D; Konecki, M; Pihir, I: Predictive Modelling of Academic
Performance by Means of Bayesian Networks, 47th International
Scientific Conference on Economic and Social Development.
• Oreški, D; Pihir, I; Konecki, M., CRISP-DM process model in
educational setting, 20th International Scientific Conference on
Economic and Social Development

12.
Research papers
• Oreški, D; Konecki, M; Milić, L., Estimating profile of successful IT
student: data mining approach, MIPRO 2017 - 40th International
Convention Proceedings
• Kovač, R; Oreški, D., Educational Data Driven Decision Making:
Early Identification of Students at Risk by Means of Machine
Learning, Proceedings of CECIIS 2018 /
• Oreški, D.; Hajdin, G., Exploring differences in predictors of
academic success between different generations of students,
EDULEARN19 Proceedings, 2019.

13.
Research papers
• Oreški, D., Hajdin, G. Development and comparison of predictive
models based on learning management system data // WSEAS
TRANSACTIONS on INFORMATION SCIENCE and APPLICATIONS,
2019
• Oreški, D.; Hajdin, G., A Comparative Study of Machine Learning
Approaches on Learning Management System Data, 2019
Proceedings - 3rd International Conference on Control, Artificial
Intelligence, Robotics & Optimization
• Filipović, D.; Balaban, I.; Oreški, D., Cluster analysis of students’
activities from logs and their success in self-assessment tests,
Proceedings of CECIIS 2018
• Oreški, D.; Kadoić, N., Analysis of ICT students' LMS engagement
and sucess, International Scientific Conference on Economic and
Social Development, 2018.

14.
Domain understanding
• Research goals:
 To determine whether data from different sources (surveys, e-
learning systems...) can be a good basis for creating predictive
models of academic success.
 To determine which variables are the best predictors of
academic success.
 To determine whether predictors of success change over time.

15.
Domain understanding
• The improvement of the educational system and the
achievement of students optimal learning requires the
data collection and analysis.
 Recent papers deal with this topic from the perspective of:
 (i) the various academic and non-academic factors involved
in the data,
 (ii) the research methodology used in data analysis:
previously focused on advanced statistical approaches,
nowadays on machine learning approaches,
 iii) accuracy and reliability of developed predictive models.

16.
Data understanding
• Data sources:
 Survey,
 Learning management system data,
 YouTube analytics.

17.
Data understanding
• Data sources:
 Survey,
 Learning management system data,
 YouTube analytics.

18.
Data understanding
Students through generations

19.
Data understanding
Students by entrance exam results
Bottom 10%-30% Middle Top 10%

20.
Data understanding
First grade at the Faculty

21.
Data understanding
I manage time well
Completely Disagree Neither agree Agree Completely
disagree or disagree agree

22.
Data understanding
I see myself as responsible person
Completely Disagree Neither agree Agree Completely
disagree or disagree agree

23.
Data understanding
I find teamwork useful
Completely Disagree Neither agree Agree Completely
disagree or disagree agree

24.
Data understanding
I have prepared for the classes
Completely Disagree Neither agree Agree Completely
disagree or disagree agree

25.
Data understanding
• Data sources:
 Survey,
 Learning management system data,
 YouTube analytics.

26.
Data understanding

27.
Data understanding

28.
Data understanding

29.
Data understanding

30.
Data understanding
• Data sources:
 Survey,
 Learning management system data,
 YouTube analytics.

31.
Data understanding
Variable Variable Correlation
Video duration Percentage of video views -0,78
Variable Variable Correlation
Complexity
Percentage of video
views
-0,35

32.
Modelling
• Information based machine learning
 Decision tree
• Similarity based machine learning
 K-nearest neighbours
 “When I see a bird that walks like a duck and swims like a duck
and quacks like a duck, I call that bird a duck.” James W. Riley
• Error based machine learning
 Neural networks
 “Success is stumbling from failure to failure with no loss of
enthusiasm.” Winston Churchill
• Probability based machine learning
 Bayesian networks
 “When my information changes, I alter my conclusions. What
do you do, sir?” John Maynard Keynes

33.
Agenda
• Introduction
 Education data & WHY?
 machine learning algorithms
• Research motivation
• Research methodology HOW?
 CRISP DM standard
• Research results WHAT?
 Model evaluation
 Model interpretation
• Conclusion

34.
Research results
rang A rang B rang C
Highschool grade
average 1 1 4
Lecture attendance 2 4 10
First grade at the
Faculty 3 3 3
I manage my time
well 4 15 13
Entrance exam
results 5 2 1
I find myself as
responsible person 8 10 5
Seminars attendance 9 11 7
I have prepared for
the classes 12 13 12
I find teamwork
useful 15 8 9
Gender 16 7 16

35.
Research results
• Strong positive correlation between predictors of academic success
in Generations B and C(r=0.652941176, p<0,01).
• Correlations in predictors between generations A and B
(r=0.370588235, p<0,01), A and C(r=0.388235294, p<0,01) are
smaller.
• Conclusion:
 Predictors of student success change over time, but only when
there are significant changes in the education system.

36.
Research results
• Female students are more active on the e-learning
system and complete the course more successfully than
male colleagues.
• Activity on the e-learning system is a significant
predictor of student success.
 Students were most active during the weeks of the colloquium,
and especially the day before the colloquium.
 Students can be characterized as "last minute" students,
because they fulfill their obligations as late as possible in terms
of the deadline.
 They are active in the "late" hours.

37.
Conclusion
• Machine learning algorithms provide accurate and
reliable predictive models.
• However, results are not perfect:
 Predicting students at risk will always suffer from classification
errors - false positives and false negatives.
 The error affects the allocation of available resources.
 Potentially negative effects on the individual.

38.
Conclusion
• To determine whether data from different sources
(surveys, e-learning systems...) can be a good basis for
creating predictive models of academic success.
 Yes! Integration contributes to successful prediction.
• To determine which variables are the best predictors of
academic success.
 First grade at the faculty, previous knowledge..
 LMS activity..
• To determine whether predictors of success change
over time.
 Change according to change in educational system.

39.
Thank you!