Presentation at AECT 2016, Las Vegas, NV

1. Digital Badges and Learning Analytics
to Explore the Learner Participation
in a MOOC
Jiyae Bong
Doctoral candidate of Instructional Systems & Learning Technologies, Florida State University
jb11ab@my.fsu.edu
Dr. Vanessa Dennen
Professor of Instructional Systems & Learning Technologies, Florida State University
vdennen@fsu.edu, @vdennen 1

2. Research Background
• The positive effects of using digital badges on the completion of learning
tasks and learners’ motivation.
• There has been a lack of empirical data on the use of digital badges and its
effects on learning engagement.
• Analytic tools embedded in MOOC platforms capture sizable data sets.
• The current study addresses this gap by applying embedded learning analytic
tool to gain data related to learners’ viewing and posting behaviors.
2

3. Research Questions
• How frequently did participants who earned badges access the main
learning content across each week of the MOOC?
• How frequently did participants who earned badges engage in main
learning activities(discussion, project, quiz) across each week of the
MOOC?
• Is there a difference between the badge recipients and non-badge
achievers on them?
• This study also looked at relationships between number of badges earned
and overall participation patterns.
3

4. About the MOOC
• Topic: Social media for active learning
• Length: 4 weeks
• Focus: Professional development
• Format: Weekly modules, Video lectures, webinars, discussions, quizzes,
& assignments
• Badge criteria: (1) participate in the discussion with at least three posts,
(2) earn 80% or higher on the quiz, and (3) complete a project
4

5. Participants
• A total of 382 participants who participated in at least one or more of
the main learning activities during the designated course period.
5
Module 1 Badge Module 2 Badge Module 3 Badge Module 4 Badge # Participants
v 20
v v 8
v v 1
v v v 3
v v v 1
v v v v 24
57 36 29 26 57
Table 1
The Number of Earned Completion Badges

6. Data Collection and Analysis
• User activity reports at the class level and the individual-learner level
including
• (a) the total numbers of access to each component of course (e.g. content area,
discussion board)
• (b) the same data rearranged by date, by hour of day, and by day of week
• Independent sample t-test (Welch-Satterthwaite version) was conducted.
• The data were rearranged and visualized considering the digital badge
achievement and course design (i.e., modularized lessons).
6

7. Findings (1) The Degree of Learner Engagement by Badge Achievement
7
Content Area:
Access (#clicks)
Discussion:
#posts
Project:
#posts
Quiz:
Score
Badge Recipients
(N=56)
33.45**
(30.41)
3.88**
(1.40)
2.43**
(1.55)
94.52*
(5.88)
Non-Badge Achievers N=201 6.58**
(6.95)
N=193 .39**
(1.16)
N=193 .41**
(3.70)
N=37 80.84*
(26.93)
Module 1 (Mean, SD)
*p<.05, **p<.001
Content Area:
Access (#clicks)
Discussion:
#posts
Project:
#posts
Quiz:
Score
Badge Recipients
(N=35)
36.74**
(18.85)
3.57**
(1.31)
1.54**
(.98)
94.71
(7.58)
Non-Badge Achievers N=206 5.36**
(13.79)
N=197 .26**
(1.01)
N=197 .05**
(.28)
N=27 88.33
(15.19)
Module 2 (Mean, SD)
**p<.001

8. Findings (1)(cont.) The Degree of Learner Engagement by Badge Achievement
8
Content Area:
Access (#clicks)
Discussion:
#posts
Project:
#posts
Quiz:
Score
Badge Recipients
(N=29)
35.66**
(24.66)
3.86**
(.13)
2.14**
(1.43)
97.33
(5.51)
Non-Badge Achievers N=229 1.99**
(8.3)
N=229 .13**
(.73)
N=229 .08**
(.84)
N=10 97
(9.47)
Module 3 (Mean, SD)
**p<.001
Content Area:
Access (#clicks)
Discussion:
#posts
Project:
#posts
Quiz:
Score
Badge Recipients
(N=26)
22.04**
(11.73)
4**
(1.94)
1.5**
(.86)
95.77
(6.28)
Non-Badge Achievers N=232 1.06**
(3.31)
N=232 .05**
(.37)
N=232 0**
(0)
N=6 81.67
(29.44)
Module 4 (Mean, SD)
**p<.001

9. Findings (2)
9
The relationships between
number of badges earned and
overall participation patterns
• The first bar graph shows
• the average numbers of clicks.
• The second and third bar graphs show
• the average numbers of posts.
• The more they earned, the more they
engaged?
• Across the modules, four-badge achievers
showed their consistent engagement level
which is similar to the badge requirements.

10. Conclusions and Implications
• The participants who earned badges consistently accessed each activity area across
the modules. Also, badge owners appeared to have a deeper level of engagement in
the required learning activities as noted by their viewing and posting behaviors.
• However, more research is required to explore the relationships between badges and
engagement in MOOCs.
• Another challenge relates to learner engagement metrics in MOOCs.
• The numerical data from clickstream analytics do not deeply explore learning. It
requires huge demands to extract the meaningful information from the numerical
data after exploring all the details of the course context.
10

11. Findings (3) The Degree of Learner Engagement by Badge Achievement
11
Content Area:
Access (#clicks)
Discussion:
#posts
Project:
#posts
Quiz:
Score
Badge Recipients
(N=56)
33.45**
(30.41)
3.88**
(1.40)
2.43
(1.55)
94.52*
(5.88)
Non-Badge Achievers
“Active” (n=36)
13.97**
(9.57)
1.06**
(1.99)
1.86
(8.46)
80.58*
(27.27)
Module 1 (Mean, SD)
*p<.05, **p<.001
Content Area:
Access (#clicks)
Discussion:
#posts
Project:
#posts
Quiz:
Score
Badge Recipients
(N=35)
36.74
(18.85)
3.57**
(1.31)
1.54**
(.98)
94.71
(7.58)
Non-Badge Achievers
“Active” (N=23)
25.09
(32.58)
1.52**
(2.27)
.39**
(.72)
87.61
(15.43)
Module 2 (Mean, SD)
**p<.001

12. Findings (3)(Cont.) The Degree of Learner Engagement by Badge Achievement
12
Content Area:
Access (#clicks)
Discussion:
#posts
Project:
#posts
Quiz:
Score
Badge Recipients
(N=29)
35.66
(24.66)
3.86*
(.13)
2.14
(1.43)
97.33
(5.51)
Non-Badge Achievers
“Active” (N=10)
16.2
(31.82)
1.9*
(2.6)
1.3
(3.46)
97
(9.48)
Module 3 (Mean, SD)
Content Area:
Access (#clicks)
Discussion:
#posts
Project:
#posts
Quiz:
Score
Badge Recipients
(N=26)
22.04
(11.73)
4*
(1.94)
1.5**
(.86)
95.77
(6.28)
Non-Badge Achievers
“Active” (N=4)
15
(13.24)
.75*
(1.5)
0**
(0)
81.25
(37.5)
Module 4 (Mean, SD) *p<.05, **p<.001
*p<.05, **p<.001

13. Thank you! Any Questions?
Contact:
Jiyae Bong – jb11ab@my.fsu.edu
Vanessa Dennen - vdennen@fsu.edu - @vdennen
13