CollabTech2014 (http://www.collabtech.org/) - This paper describes the analysis of collaborative mobile learning activities. We explore the use of learning analytics for the evaluation of the performance of stu-dents as individuals and the performance of teams. We argue that traditional met-rics used for learning analytics can provide insight with respect to the quality of the activity and the learning outcome. We propose a way to integrate innovative mobile learning scenarios into traditional classrooms and to analyze collaborative learning activities on both the group and the individual level.

1. Multilevel analysis of collaborative activities
based on
a Mobile Learning Scenario
for real classrooms
Irene-Angelica Chounta, Adam Giemza, H. Ulrich Hoppe
Collide, University of Duisburg-Essen
{chounta, giemza, hoppe}@collide.info

2. Mobile learning: a special case of learning....
Learning on the move, across space, within context
Activities supported by mobile devices
- „location-based“ scenarios with spatially distributed
learning stations (museum, outdoor trips etc)
- smartphones, tablets etc.
The activity analysis is carried out with qualitative
methods

3. Objectives of the study
Introduction of a mobile learning scenario in
traditional school classrooms
Use of common learning analytics within a
mobile learning context
Study of collaborative learning activities on
multiple levels of analysis

4. a mobile learning scenario for the classroom
Computer Kit: „Learn how to assemble a
Personal Computer with Mobilogue“

5. a mobile learning scenario for the classroom
QR codes as learning station/ computer parts
markers in a very localized setting
Learning Content Video Quiz Scores

6. Mobilogue Architecture
Authoring environment for the creation of new
activities
Activity logs for recording and post analysis
Online Management Environment

7. Method of the study
Two case studies:
- Case A: preliminary study, 17 students
- Case B: main study, 24 students
13 – 15 years old
Task: Group of students working together to assemble
a pc
Each group was supported by one mobile device
The activities were analyzed with respect to:
- students practice on the individual and the group level
- group characteristics
- learning outcome

8. Method of the study
For the analysis we used:
- Activity metrics from logfiles
- Average time gap among consequent actions
- Average response time to quizzes
- Quiz score
- Experts observations
- Activity transcripts (recorded by students)
- Users questionnaires (with respect to user experience
and motivation for collaboration)
- Pre and Post Knowledge Tests (each test consisted of
a ten-items questionnaire)

9. Case study A - Results
17 students randomly grouped in 5 groups
Activity duration ≈ 90 minutes
Group profiles based on pre-tests:
One „strong“ team (team A)
One „weak“ team (team E)
Three average teams (teams B, C, D)
Pretest Scores
avg_group max_idv
teamA 6.25 8
teamB 3.75 5
teamC 4.33 5
teamD 2.67 4
teamE 1 1
heterogeneous
homogeneous

10. Case study A - Results
Greatest improvement teams of average performance (on
group and individual level)
The “strong” team maintained the same score on average
but scored less on the individual level - homogeneity
increased)
The “weak” team slightly improved – heterogeneity
increased
Heterogeneous teams increase homogeneity and vice versa
Pre-test Scores
avg_group max_idv
teamA 6.25 8
teamB 3.75 5
teamC 4.33 5
teamD 2.67 4
teamE 1 1
Post-test Scores
avg_group max_idv
6.25 7
6.25 8
5.67 7
6.00 7
2.00 3

11. Case study A - Results
Quiz score reflects the knowledge test performance
The average time gap correlates negatively with quiz
score
Teams with good knowledge background and high
heterogenity move faster
Pre-test Scores
avg_group max_idv
teamA 6.25 8
teamB 3.75 5
teamC 4.33 5
teamD 2.67 4
teamE 1 1
Activity metrics
avg_timegap
avg
response
time
Quiz
score
27.63 32.02 120
36.56 56.96 120
37.14 23.70 100
37.91 31.55 90
38.43 57.43 80

12. Case study B - Results
24 students – 6 groups of four
Groups were created by the teacher
Activity duration ≈ 90 minutes
Special Additions / Modifications:
Experts monitored the activity
Students kept transcripts of group activity
Students filled in questionnaires regarding their
experience on collaboration

13. Case study B - Results
Group profiles based on pre-tests:
One strong team (team 03)
Two weak teams (team 00 and team 05)
Three average teams (teams 01, 02, 04)
The Homogeneity of teams according to pre-test
confirmed teacher‘s perception
Pre-test Scores
avg_group max_idv
team 00 2 4
team 01 5.75 9
team 02 4.25 5
team 03 6.5 8
team 04 5.5 7
team 05 1 2
heterogeneous
homogeneous

14. Case study B - Results
Group performance improved on average (47%) and
individual (31%)
Max knowledge gain weak team (team 05)
Min knowledge gain strong team (team 03)
Knowledge „loss“ for the „strong“ students
Homogenity increases for heterogenous groups and vice
versa (confirming case study A)
Pre-test Scores
avg_group max_idv
team 00 2 4
team 01 5.75 9
team 02 4.25 5
team 03 6.5 8
team 04 5.5 7
team 05 1 2
Post-test Scores
avg_group max_idv
3.5 7
7.5 8
6 7
6.5 7
7.25 8
6 9

15. Case study B - Results
Heterogeneous teams achieved higher Quiz scores
The average time gap correlates negatively with quiz
score (also observed case study A)
Heterogeneous teams move faster, i.e. Individuals take
the lead Activity metrics
avg_timegap
avg
response
time
Quiz
score
20.89 51.88 120
23.42 49.02 120
24.00 45.73 110
25.84 61.99 90
40.44 72.51 70
27.75 82.56 70
Pre-test Scores
avg_group max_idv
team 00 2 4
team 01 5.75 9
team 02 4.25 5
team 03 6.5 8
team 04 5.5 7
team 05 1 2

16. Case study B - Results
The students undertook three roles throughout the scenario:
the operator of the mobile device
the scriber and
the assembler
1.4
1.2
1
0.8
0.6
0.4
0.2
0
team00
u00-00 u00-01 u00-02 u00-03
1.4
1.2
1
0.8
Scriber
Assembler
0.6
Operator
0.4
0.2
0
team01
u01-00 u01-01 u01-02 u01-03
1.4
1.2
1
0.8
0.6
0.4
0.2
0
team02
u02-00 u02-01 u02-02 u02-03
1.4
1.2
1
0.8
0.6
0.4
0.2
0
team03
u03-00 u03-01 u03-02 u03-03
1.4
1.2
1
0.8
0.6
0.4
0.2
0
team04
u04-00 u04-01 u04-02 u04-03
1.4
1.2
1
0.8
Scriber
Assembler
0.6
Operator
0.4
0.2
0
team05
u05-00 u05-01 u05-02 u05-03
Scriber
Assembler
Operator
• 4 out of 6 teams followed no strict
role distribution
• Teams following strict role
distribution showed the lowest
knowledge dissatisfaction

17. Discussion
A mobile scenario to support collaborative activities
for traditional classrooms
Multilevel analysis of the activity (various methods
and multiple units of analysis) to enhance the overall
outcome
Learning Outcome
• The setting was more effective for students of weak
and medium knowledge background
• Strong students/ teams showed little or no knowledge
gain
• The students performance was increased on both
individual and group level

18. Discussion
Group Characteristics Learning Analytics
• Heterogeneous teams achieved higher game scores
• Game score did not correlate to knowledge gain or
knowledge background
• Heterogeneous teams moved faster through the
stages of the scenario
• Homogeneous teams tend to increase their
heterogeneity and vice versa

19. Discussion
Collaboration
• Team members undertook roles with no particular
plan
• In the case a strict plan was follows, it lead to
frustration and low knowledge gain
• Strong students stated they would have done better
on their own
• Weak/average students considered the activity as
helpful

20. Conclusion and Future Work
Multilevel analysis can provide valuable information
and enhance mobile learning
Extend the field of study to further investigate:
the use of various scenarios
varying degrees of freedom to classroom activities
concurrent use of multiple devices within a team
the nature of learning activities that could be
supported effectively by the use of mobile devices
ways to incorporate seamlessly mobile devices in
traditional learning activities

21. Questions.
Thank you ☺
Thank you ☺
chounta@collide.info