Slides from our presentation at LAK'18 Conference in Sydney, NWS. The full paper is available at http://dx.doi.org/10.1145/3170358.3170374

1. 1
Understand students’ self-reflections
through learning analytics
Vitomir Kovanović¹, Srećko Joksimović¹, Negin Mirriahi1
Ellen Blaine2, Dragan Gašević3,4, George Siemens5, and Shane Dawson1
¹ The University of South Australia, Australia
² Stanford University, USA
³ Monash University, Australia
⁴ University of Edinburgh, United Kingdom
5 University of Texas at Arlington, USA
March 9, 2018
Sydney, NSW, Australia
Vitomir.Kovanovic@unisa.edu.au
http://vitomir.kovanovic.info
@vkovanovic

2. 2
Reflection and Self-regulated Learning
Student self-reflection:
• A key strategy for practicing critical thinking
• Develop autonomy, and confidence in their learning
• Common approach for development of self-regulation skills
Common self-reflection methods:
• Writing reflective journals
• Reflective video annotations

3. 3
COPES model of Self-regulation
(Butler and Winnie, 1995)

4. 4
Reflection and Self-regulated Learning
Depth of reflections matters:
• Providing observations & commentaries
• Describing motivations behind certain behaviors
• Setting goals for practice
Prior work:
• Different depth  Different learning outcomes
• Prior experience with reflective activities matter (Gašević et al., 2017)
• Scaffolding and support vital for student success and effectiveness of
reflective activities

5. 5
Reflection and Self-regulated Learning
• Providing feedback & guidance essential but time consuming!
• Research on student-reflection is also complicated and expensive
(content analysis)
• Hard to provide real-time feedback & monitoring of student progress
Previous work focused on reflection in essays (e.g., Ullmann et al 2013;
Ullmann, 2015; Gibson and Kitto, 2015; Gibson et al. 2016)
Three key approaches:
1. Dictionary-based
2. Rule-based
3. Machine-learning-based

6. 6
Overall goal
and why it matters
Automate the analysis of students’ self-reflective
video annotations
RQ1: What are the indicators of self-reflection?
RQ2: Can we do it (accurately enough)?

7. 7
Overall goal
and why it matters
Automate the analysis of students’ self-reflective
video annotations
Use psychologically-sound and established
measures of psychological processes

8. 8
Collaborative Lecture Annotation System (CLAS)
Video annotation software

9. 9
Open Video Annotations for Learning (OVAL)
Essentially CLAS 2.0

10. 10
Data: Courses
Four UG courses in performing arts offered in the 12-13 academic year at a
large research-intensive public university in Canada.
Course Recording
type
CLAS
required
Enrolled
students
Coded units of
analysis
Course 1 Group No 31 145 (3.27%)
Course 2 Individual Yes 40 1,393 (31.44%)
Course 3 Individual Yes 28 2,457 (55.46%)
Course 4 Individual No 20 435 (9.82%)
Total: 119 (771) 4,430 (100.00%

11. 11
Reflection coding scheme
• A modified version of Hulsman et al. (2009) coding scheme
• Annotations split into units of analysis (utterances)
• Two coders codes subset of data until Cohen’s K of 0.75 reached
Category Definition Example
Observation Student indicates what they
observed about their own
behavior, but does not indicate
why the behavior occurred.
“I still continue to have
problems making eye
contact...”
Motive Student indicates what they
observed and why it occurred.
“...being up there made me
insecure and nervous, which
led to my eyes dropping
frequently...”
Goal Student indicates what they will
do next time or what they need
to work on.
“What I really want to avoid
is ending up just mirroring
everything.”

12. 12
Text classification
Build a classifier for video self-reflection
By automating coding of video annotations, we can
overcome many challenges of video reflection assessment.
Builds on previous text-mining work in education
We build on the previous work of Kovanovic et al. (2016)
Combine N-grams with linguistic metrics
To provide both context-free and context-specific
indicators of reflection
Four-class text classifier
Classifier needs to categorize each message to:
1-Observation, 2 – Motive, 3 – Goal, 0 - Other.

13. 13
Train-test split
• Data split into 75%–25% train and test datasets
Dataset
Category Train (75%) Test (25%) All
Observation 1,135 (34.17%) 382 (34.48%) 1,517 (34.24%)
Motive 1,848 (55.63%) 625 (56.41%) 2,473 (55.82%)
Goal 174 (5.24%) 56 (5.05%) 230 (5.19%)
Other 165 (4.97%) 45 (4.06%) 210 (4.74%)
Total: 3,322 (100%) 1,108 (100%) 4,430 (100%)

14. 14
Extracted features
503 features in total extracted
1 3
2 4
LIWC features
93 measures of different psychological processes
N-grams
Top 100 unigrams, bigrams, and trigrams
Coh-Metrix features
109 metrics of text coherence (and related metrics)
Context features
First unit in sentence (binary feature)

15. 15
Features: Top 10 N-grams
Unigram Freq. Bigram Freq. Trigram Freq.
need 383 left hand 112 practice front mirror 17
conducting 279 eye contact 71 use left hand 14
think 248 need work 55 third goal would 11
music 239 make sure 54 make eye contact 10
really 200 front mirror 36 second goal would 10
hand 182 goal would 32 first goal would 10
practice 181 feel like 30 three critical goals 8
ensemble 171 beat pattern 30 critical goals improvement 8
work 170 right hand 29 really need work 8
beat 161 also need 26 influence sound moment 7

16. 16
SMOTE preprocessing
SMOTE preprocessing for class balancing. Dark blue
– original instances which are preserved, light blue – synthetic instances
We generate new data points in minority classes by “syntactic resampling”
using SMOTE technique.
To generate a new data point (Z) ∈ Rn:
• Pick a random data existing data point (X),
• Pick K (in our case 5) instances most similar to the given data point,
• Pick randomly one of the K neighbors (Y)
• Create a new data point Z as a linear combination: Z = X + rand(0,1)*Y

17. 17
Random Forest classifier
A state-of-the-art ensemble learning method:
• Builds a large collection of decision trees (i.e.,
forest) using a subset of features (i.e., columns)
• Reduces the variance without increasing the bias
• Final class for a data point: a simple majority vote
across the forest.
Two parameters:
1. ntree – the number of trees built
2. mtry – number of features used
ntree = 6

18. 18
Random Forest classifier
Individual tree
mtry=3
8 features

19. 19
Implementation
Feature Extraction
• N-grams; NLTK Python library
• Coh-Metrix (McNamara, Graesser, McCarthy, & Cai, 2014)
• LIWC (Tausczik & Pennebaker, 2010)
Algorithm implementation
• Stratified sampling implemented using scikit-learn Python library
• SMOTE algorithm implemented using unbalanced R package
• Random Forest classifier using randomForest R package
• Repeated cross-validation using carret R package

20. 20
Hyper-parameter tuning
1. ntree – the number of trees built
(we built 500)
2. mtry – number of features used
(evaluated using 10-fold CV)
Values examined:
{2, 3, 4, 6, 8, 11, 15, 20, 27, 36, 48,
65, 87, 156, 209, 279, 373, 500}
mtry Accuracy (SD) Kappa (SD)
Min 2 .81 (.01) .75 (.02)
Max 116 .89 (.01) .85 (.02)
Difference .07 .10
Two parameters optimized
Hyper parameter tuning results
Hyper parameter tuning

21. 21
Performance evaluation
We obtained:
• 75% classification accuracy
95% CI[0.72, 0.77]
• 0.51 Cohen’s κ
Predicted
Other Observation Goal Motive Error rate
Actual
Other 9 9 27 0 .80
Observation 1 250 131 0 .34
Goal 1 59 564 1 .10
Motive 0 30 25 1 .98
Confusion matrix
Out-of-bag (OOB) error rate

22. 22
Performance evaluation
• “Moderate” level of agreement (Cohen, 1988)
• Parameter optimization important (.1 Cohen’s κ difference, 7%
classification accuracy).
• Best-performance for two most common classes
• Features both encode general properties of self-reflection and particular
learning context
• “Theory-driven” context-free features
• Simple context-specific features

23. 23
Feature Importance
• A side product of Random Forest algorithm
• Mean Decrease Gini (MDG) measure of
feature contribution to reducing decision
tree impurity
• A long tail of feature importance
• Few features very important, most not so
much
• Provides more detailed insights into what
characterizes different types of self-
reflection
Min Q1 Median Mean Q3 Max.
0.00 0.12 1.06 10.82 15.34 219.94
Summary of classification feature importance

24. 24Feature importance
Category
# Variable Description MDG Other Obser. Goal Motive
1 liwc.see Perceptual processes: seeing 220 1.01 1.64 1.62 1.55
2 cm.SMCAUSr Situation model: ratio of casual particles to verbs 201 0.12 0.13 0.11 0.37
3 cm.DRPVAL Agentless passive voice density, incidence 184 2.46 3.02 2.03 4.66
4 liwc.focuspast Time orientation: focus towards past 152 1.46 4.57 0.80 4.80
5 cm.WRDNOUN Word information: noun incidence 121 252 186 208 194
6 liwc.ingest Biological processes: ingestion (e.g., dish, eat) 112 0.58 0.30 0.33 0.39
7 cm.CNCCaus Causal connectives, incidence 105 28.27 20.86 28.57 42.91
8 trust.ensemble Freq. of “trust ensemble” bigram 95 0.00 0.00 0.00 0.00
9 cm.SMINTEr Ratio: intentional particles to int. actions/events 88 0.30 0.23 0.36 0.47
10 liwc.Period Punctuation: use of full stop 75 9.17 6.38 7.14 5.45
11 cm.DRNP Syntactic pattern den: incidence of noun phrases 72 391 320 306 314
12 chamber.music Freq. of “chamber music” bigram 71 0.00 0.00 0.00 0.01
13 liwc.AllPunc Punctuation: all (e.g., periods, commas) 69 19.22 13.02 14.06 11.66
14 liwc.cause Cognitive processes: causality (e.g., because) 63 2.02 1.85 2.03 3.98
15 cm.SMCAUSv Situational model: incidence of causal verbs 61 33.60 46.08 65.65 44.28
16 liwc.insight Cognitive processes: insight (e.g., think, know) 60 2.98 3.64 2.36 3.50
17 cm.SYNLE Syntactic complexity: mean number of words before
the main verb in the main clause
58 2.36 2.65 1.78 3.53
18 liwc.home Personal concerns: home (e.g., kitchen, landlord) 56 0.10 0.08 0.05 0.04
19 liwc.Analytic Summary measures: formal, logical, hier. thinking 52 60.02 54.60 70.94 57.36
20 liwc.percept Perceptual processes: all (e.g., look, heard) 52 4.20 4.85 4.69 5.04

25. 25Feature importance
Category
# Variable Description MDG Other Obser. Goal Motive
1 liwc.see Perceptual processes: seeing 220 1.01 1.64 1.62 1.55
2 cm.SMCAUSr Situation model: ratio of casual particles to verbs 201 0.12 0.13 0.11 0.37
3 cm.DRPVAL Agentless passive voice density, incidence 184 2.46 3.02 2.03 4.66
4 liwc.focuspast Time orientation: focus towards past 152 1.46 4.57 0.80 4.80
5 cm.WRDNOUN Word information: noun incidence 121 252 186 208 194
6 liwc.ingest Biological processes: ingestion (e.g., dish, eat) 112 0.58 0.30 0.33 0.39
7 cm.CNCCaus Causal connectives, incidence 105 28.27 20.86 28.57 42.91
8 trust.ensemble Freq. of “trust ensemble” bigram 95 0.00 0.00 0.00 0.00
9 cm.SMINTEr Ratio: intentional particles to int. actions/events 88 0.30 0.23 0.36 0.47
10 liwc.Period Punctuation: use of full stop 75 9.17 6.38 7.14 5.45
11 cm.DRNP Syntactic pattern den: incidence of noun phrases 72 391 320 306 314
12 chamber.music Freq. of “chamber music” bigram 71 0.00 0.00 0.00 0.01
13 liwc.AllPunc Punctuation: all (e.g., periods, commas) 69 19.22 13.02 14.06 11.66
14 liwc.cause Cognitive processes: causality (e.g., because) 63 2.02 1.85 2.03 3.98
15 cm.SMCAUSv Situational model: incidence of causal verbs 61 33.60 46.08 65.65 44.28
16 liwc.insight Cognitive processes: insight (e.g., think, know) 60 2.98 3.64 2.36 3.50
17 cm.SYNLE Syntactic complexity: mean number of words before
the main verb in the main clause
58 2.36 2.65 1.78 3.53
18 liwc.home Personal concerns: home (e.g., kitchen, landlord) 56 0.10 0.08 0.05 0.04
19 liwc.Analytic Summary measures: formal, logical, hier. thinking 52 60.02 54.60 70.94 57.36
20 liwc.percept Perceptual processes: all (e.g., look, heard) 52 4.20 4.85 4.69 5.04

26. 26
Characteristics of different self-reflection statements
Unique characteristics
O
Observation
Use of perception words
Oriented towards past
Low causal connectiveness
Low levels of higher order critical thinking
M
Motive
High ratio of causal particles to verbs
Frequent use of passive voice (agentless)
Oriented towards past
Expressing causality
Complex verb phrases
G
Goal
Use of perception words
Low expression of insight
Simple verb phrases
High levels of higher order critical thinking
Frequent use of causal verbs
Non-reflective (other)
Low use of perception words
Express of personal matters
High use of punctuation
High use of noun phrases

27. 27
Summary
Almost done
• We developed a classifier for automated coding of students’ in-video annotations with respect to
the expressed
• Our exploratory analysis is encouraging. We achieved 75% percent accuracy, Cohen’s k=.51
• The adopted features both capture context-free and also context-specific nature of self-reflection
• We provided more detailed operationalization of the different types of reflective statements
Future work:
• Replicate the study using the new data from other course (use existing model on the new course)
• Examine additional features that better capture goal reflective statements

28. 28
The end
That is all folks
Thank you

29. 29
References (I/II)
Butler, D. L., & Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis.
Review of Educational Research, 65(3), 245–281. https://doi.org/10.3102/00346543065003245
Cohen, J. (1988). The Analysis of Variance. In Statistical power analysis for the behavioral sciences
(pp. 273--406). Hillsdale, N.J.: L. Erlbaum Associates.
Gašević, D., Mirriahi, N., Dawson, S., & Joksimović, S. (2017). Effects of instructional conditions and
experience on the adoption of a learning tool. Computers in Human Behavior, 67, 207–220.
https://doi.org/10.1016/j.chb.2016.10.026
Gibson, A., & Kitto, K. (2015). Analysing Reflective Text for Learning Analytics: An Approach Using
Anomaly Recontextualisation. In Proceedings of the Fifth International Conference on Learning
Analytics and Knowledge (pp. 275–279). New York, NY, USA: ACM.
https://doi.org/10.1145/2723576.2723635
Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the Discovery of Learner Metacognition From
Reflective Writing. Journal of Learning Analytics, 3(2), 22–36.
https://doi.org/10.18608/jla.2016.32.3

30. 30
References (II/II)
Hulsman, R. L., Harmsen, A. B., & Fabriek, M. (2009). Reflective teaching of medical communication
skills with DiViDU: Assessing the level of student reflection on recorded consultations with
simulated patients. Patient Education and Counseling, 74(2), 142–149.
https://doi.org/10.1016/j.pec.2008.10.009
McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated Evaluation of Text
and Discourse with Coh-Metrix. Cambridge University Press.
Tausczik, Y. R., & Pennebaker, J. W. (2010). The Psychological Meaning of Words: LIWC and
Computerized Text Analysis Methods. Journal of Language and Social Psychology, 29(1), 24–54.
https://doi.org/10.1177/0261927X09351676
Ullmann, T. D. (2015). Automated detection of reflection in texts. A machine learning based
approach (PhD thesis). The Open University.
Ullmann, T. D., Wild, F., & Scott, P. (2013). Comparing automatically detected reflective texts with
human judgements. In 2nd Workshop on Awareness and Reflection in Technology- Enhanced
Learning (pp. 101–116). Retrieved from http://oro.open.ac.uk/id/eprint/37830