1. Learning Analytics Applied to Serious Games
European Conference in Game-Based Learning (ECGBL
2013), Porto, Portugal
Baltasar Fernandez-Manjon, balta@fdi.ucm.es , @BaltaFM
e-UCM research group, www.e-ucm.es

2. About me and context
‣ CS Professor at Complutense U.
• Director of e-UCM
‣ e-UCM research group about
Learning technologies
• 15 researchers
• Serious games
- Application to the medical domain
• European projects
- GALA
- SEGAN
- CHERMUG
• www.e-ucm.es
2

3. Learning analytics and SG?
The NMC Horizon Report:
2013 Higher Education
Edition
‣ new and emerging
technologies on
teaching, learning, and
research
Time-to-Adoption
Horizon: Two to Three
Years
‣ Games and
Gamification
‣ Learning Analytics
3
http://www.nmc.org/publications/2013-horizon-report-higher-ed

4. Serious Games use?
‣ Serious Games have probed to be educationally
effective in several domains
• Medicine, military, business, corporate training
‣ But still is a low adoption of Serious Games
• Cost? ROI?
‣ Serious Games considered usually as a
complementary content
• Mainly used for motivational purposes
• No actual impact on the final mark
‣ Difficult to include Serious Games in the learning
curriculum
• Assessment of acquired learning?
4

5. Black box model
‣ Games as independent
pieces of content
‣ No information about
what is happening
during the in-game
play
‣ Or very simple
• Completed or not
completed
• Time used
5
del Blanco et al (2013). Using e-Learning standards in educational video games.
Computer Standards & Interfaces 36 (1) pp. 178–187

6. Learning Analytics
‣ Improving education
based on data
analysis
• Data driven
‣ Evidence-based
education
‣ Related with …
• Educational data mining
• Business intelligence
• Visual analytics
6
www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf

7. Different uses
‣ Learning analytics
• Data analysis that helps students improve learning
outcomes.
‣ Academic/program analytics
• Data analysis that provides information of what is
happening in a specific program and how to plug holes
or otherwise adjust.
‣ Institutional analytics
• Data analysis that helps make decisions about how to
improve at the institutional level.
7
Learning Impact Blog, Big data: Cool; Small data: Cooler http://www.imsglobal.org/blog/?p=258

8. When data is analyzed?
‣ Off-line
• Analyzing data after use
• Discovering patterns of use
• Allows to improve the experience for future
‣ Real time
• Analyzing data while the system is in use to
improve/adapt the current learning experience
• Allows to also use it in actual presential classes
‣ Mixed approach
8

9. Extensive Data
‣ Large number of participants
‣ Relatively limited number of variables
‣ Usually very little demographic information
‣ Relatively few observations for each user
‣ Wide but shallow data set
9
Adapted from: Learning Analytics and Educational Data Mining Workshop
New York University – CREATE Lab April 4–5, 2013

10. Intensive Data
‣ Relatively low number of participants
‣ Large number of observations for each variable
‣ Large number of variables for each participant, such
as
• User actions, In-Game Events
• Survey responses; Extensive demographic information
• Video Observations
• Biometric Data (HR, RESP, GSR, EEG, EKG)
• Eye-tracking
‣ Narrow but deep data set
‣ Correlations among different data?
10
Adapted from : Learning Analytics and Educational Data Mining Workshop
New York University – CREATE Lab April 4–5, 2013

11. But there is a parallel world …
11
http://us.battle.net/wow/en/media/screenshots/races?keywords=&view#/goblins04

12. Game Analytics
‣ Application of
analytics to game
development and
research
‣ Telemetry
• Data obtained over
distance
• Mobile games, MMOG
‣ Game metrics
• Interpretable
measures of data
related to games
12

13. Game metrics
‣ User metrics
• Customer
• Player
‣ Performance
metrics
• Technical
infrastructure
‣ Process metrics
• Development of the
game
‣ User metrics
• Generics metrics
• Genre specific metrics
• Game specific metrics
13

14. Game metrics
14
Springer, Game Analytics Maximizing the Value of Player Data, pag 22

15. Game requirements for LA
‣ Most of games are black boxes.
• No access to what is going on during game play.
‣ We need access to game “guts”
• Game state, game variables
‣ Or the game must communicate with the outside
world
• Using some logging framework
• Not applicable to COTS games (yet)
‣ Mozila Open Badges? http://openbadges.org/
15

16. SESSION
VARIABLES
PLAYER
GAME
VARIABLES
GAME
VARIABLES
TRACES
GOALS
From game data to educational information
PLAYER
VARIABLES
User Information
Educational System Information
SERIOUS GAME

17. Sessions
GAME PLAYER SESSIONS+ =
A player plays a game. Produces a game session.
One game session is a set of traces of ONE specific
player in ONE specific game.
SESSIONS =
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
player
started
phase 1
player
scored
200
in
phase 1
player
clicked
in
Help
button

18. VARIABLES
GENERATO
R
SESSION
Analyzing game sessions
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
TRAC
E
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
GOAL
RESULT
GOAL
RESULT
GOAL
RESULT
GOAL
RESULT
GOAL
RESULT
GOALS
GENERATO
R
VAR
DEFINITIO
N
VAR
DEFINITIO
N
VAR
DEFINITIO
N
VAR
DEFINITIO
N
VAR
DEFINITIO
N
VAR
DEFINITIO
N
GOAL
DEFINITIO
N
GOAL
DEFINITIO
N
GOAL
DEFINITIO
N
GOAL
DEFINITIO
N

19. Variable generator: User sessions
VAR
DEFINITION
VAR+ =
TRACE
TRACE
varName:‘beatgame‘,
operation: {
type:'trace_present‘
,traceType: ‘logic',
trace: {
event:‘end_game’
}
}
event:
‘end_game',
timeStamp:
1380181610150
usersession.beatgame
= true;+ =

20. Goal Generators: User sessions
VAR
VAR
GOAL
DEFINITION
GOAL
RESULT=
usersession: {
beatgame: true,
score: 4747
}
id: ‘finalResult’
result: ‘beatgame
&& score > 3000’
}
usersession: {
beatgame: true,
score: 4747,
goals: {
finalResult: true
}
}
=
+
+

21. PLAYER * GAME
Analyzing game results
SESSION
SESSION
VAR VAR
VAR VAR
GOAL
RESULT
GOAL
RESULT
SESSION
SESSION
VAR VAR
VAR VAR
GOAL
RESULT
GOAL
RESULT
VARIABLES
GENERATO
R
GOALS
GENERATO
R
VAR
DEFINITIO
N
VAR
DEFINITIO
N
VAR
DEFINITIO
N
VAR
DEFINITIO
N
VAR
DEFINITIO
N
VAR
DEFINITIO
N
GOAL
DEFINITIO
N
GOAL
DEFINITIO
N
GOAL
DEFINITIO
N
GOAL
DEFINITIO
N

22. eAdventure game platform
Open code authoring environment for the production of point-and-click
adventure games & immersive learning simulations
Easy to include Learning Analytics in eAdventure games

23. http://www.chermug.eu
With MGH-Harvard Universityhttp://first-aid-game.e-ucm.es
With ONT, educ@ONT

24. eAdventure + Learning
Analytics
Game Engine
Communication
API
{
type: 'input',
timeStamp: some_timestamp,
device: 'some_device',
action: 'some_action',
target: 'target_id',
data: { key1: value, ...}
}
{
type: 'logic',
timeStamp: some_timestamp,
event: 'some_event',
target: 'some_id',
data: { key1: value, ...}
}
Learning
Analytics
Database

25. GLEANER
‣ GLEANER: Game Learning Analytics for education
research
• Open code framework to capture game traces
25
Reference model in the EU NoE GALA

26. GLEANER Collector
‣ Implemented: Nodejs + Mongodb
‣ A web server ready to receive traces
‣ Cleans and sorts the data

27. GLEANER Analysis
‣ Reporter has access to the database, and presents
its data through reports
• graphics, heat-maps, relational tables…
‣ Evaluator has access to the database and checks the
educational defined goals in the assessment model

28. Example: A game to learn XML
http://gleaner.e-ucm.es/lostinspace/play/index.html

29. About the game...
‣ A basic platform game to acquire familiarity with
XML documents
• Learn the syntax
• Understand nesting and attributes
• Gain agility writing and reading XML documents
‣ Designed as a complementary activity in a Web
Programming Course
• For undergraduate computer science students
‣ Not developed with eAdventure

30. Understanding the game…
Goal
Main
character
Power-ups
(new syntax
elements)
Write XML
snippets here to
move the main
character to the
goal

31. Educational goals
‣ Basic understanding of XML Syntax
• Markup syntax
• Basic documents
• Attributes
• Complex documents
‣ Basic programing skills
• Sequencing
• Loops

32. LA perspective
‣ What does GLEANER trace?
• Higher level events
• Generic traces & game specific traces
• Only those relevant for our learning objectives
‣ The aggregator will filter and transmit:
• Level completion events
• Each XML fragment submitted by the player

33. Main dashboard (GLEANER Report)
33

34. Progress report
34
User
Session
Phases
completed
Current
score
Learning
achievements
Achievements are
updated and
highlighted in real
time

35. Detailed traces
35
Filters XML
snipets
The fragments are updated in real time
as submitted by the student. The
instructor can filter to see only mistakes,
or all the fragments submitted by a
specific students.

36. La Dama Boba : The Game
Based on The Foolish Lady by Lope de Vega
The game is available at http://damaboba.e-ucm.es/ (in Spanish)

37. Experiment game vs class
‣ 757 students
‣ From 8 middle and
high schools in Madrid
region
‣ Control group and
experimental group

39. Formal evaluation pre-post
‣ Similar results with
Learning Analytics
than with pre-post
test?
39

40. learning analytics outcomes

41. 41

42. Other issues in LA
‣ Ethical and legal aspects
‣ Security model
‣ Ownership of information
• Informing the user
‣ Anonymization of information
‣ Aspects specially relevant if you are working with
kids!
42

43. But there are new especifications
and develpments that could
sistematize the work
43

44. ADL eXperience API (xAPI)
‣ Result of Project Tin Can
‣ Tracks experiences, informal learning, real-
world experiences (not just completions)
‣ Allows data storage AND retrieval (ex. 3rd
party reporting and analytics tools)
‣ Enables tracking mobile, games, ITS, and
virtual worlds experiences
‣ Developed by open source community
44
From Damon Regan (ADL) presentation at SINTICE2013

45. 45
Activity Streams
‣ http://activitystrea.ms
‣ Collaboration between Google, Facebook, Microsoft
and others
‣ Allows reporting of experiences, not just completions
‣ Format: <Actor> <Verb> <Object> (I did this):
Simple Statement:
I (actor) watched (verb) a video on protecting employee data
(object)
Complex:
in the context of [information assurance certification
training] with result [timestamp:2013-0618T18:30:32.360Z ].
From Damon Regan (ADL) presentation at SINTICE2013

46. Reporting
Systems
Assessme
nt Services
Semantic
Analysis
Statistical
Services
xAPI Learning Record Store (LRS)
46
From Damon Regan (ADL) presentation at SINTICE2013

47. ‣ Raw data can feed several systems
• An LRS
• A Learning Analytics System
eAdventure + LA with xAPI
Raw data
LRS
Learning
Analytics
System
Statements
Analyzer
Statements
Analyzer
EXPERIENCE
API
EXPERIENCE
API

48. IMS Global Learning Analytics Interoperability
Framework
http://www.imsglobal.org/IMSLearningAnalyticsWP.pdf

50. Conclusions
‣ LA in Serious Games has a great potential from the
application and research perspective
• Simplify more complex and complete experiments
‣ LA in Serious Games should benefits from Games
Analytics experience and work
‣ Still complex to implement LA in SG
• Increases the (already high) cost of the games
‣ Frameworks and new standards specifications could
greatly simplify LA implementation and adoption
50

51. Thank you!
51
balta@fdi.ucm.es
@BaltaFM

52. Our current projects
Increasing patient safety
using games
Modelling/teaching medical procedures at
National Transplant Organization