1. A synopsis of Educational Technology R&D:
Lessons learned from a selection of theses
submitted to Open University of Cyprus
Dr. Panagiotis Zaharias
Open University of Catalonia - 4/6/2015

2. Experience from master theses supervision at
Open University of Cyprus
 Mentored/supervised more than 20 students in their
master thesis along topics such as:
 - Usability and UX methods,
 - Serious games / Game-based learning,
 - Online and video games,
 - Virtual worlds,
 - Gamification,
 - Social media in education etc.

3. Assessing long-term UX in the context of
MMOGs

4. Time spans of User Experience (UX)
Source: http://www.allaboutux.org/files/UX-WhitePaper.pdf

5. UX in the long term: an important but
neglected issue
 While the importance of temporality has been
repeatedly highlighted in user experience research, it
has rarely been systematically addressed.
 …Due to the effort involved in conducting longitudinal
studies.
 or a lack of sufficient interest, induced by a belief that
motivating prolonged use does not necessarily lead to
increased commercial revenues.

6. Context of the study
 Based on:
 Thesis submitted by Moschou, Eirini - thesis title: “Development of a UX
evaluation method for Virtual Learning Environments” - January 2014
 Why long term UX in MMOGs?
 Shortage of relevant studies so far
 Players usually have a long lasting relationship with the game they play
 The main objective of the study:
 To investigate, understand and interpret the changes of players’
experience over time

7. The method: A new version of UX Curve
 UX Curve: A retrospective method for assessing long term UX
proposed by Kujala et al. (2011).
Based on drawings made by users
Cost efficient as a long term usage
can be covered in a single session
Results give an overview of the most
relevant experiences
Memories are as important or more
than actual experiences

8. UX Curve templates

9. Context of empirical study
 The game: League of Legends
 The players: 9 students aged between 18-20 years old who had been
playing LoL for periods ranging from 6 to 16 months (mean: 12.5
months SD: 3.1 months)
 The new version of UX Curve:
UX Curve
• General UX
• Attractiveness
• Ease of use
• Utility
• Usage Volume
A new UX Curve
(MMOG version)
• General UX
• Degree of usage
• Ease of use
• Immersion
• Social interaction
• Engagement

10. Indicative results (1)
 A total of 54 curves were collected. Most of the curves were
improved revealing thus an increasing user experience
General UX Curve for each user ID The immersion UX Curve for each user ID

11. Indicative results (2)
 In order to draw the curves, players described the factors that
improved their experience over time or caused it to deteriorate
Reason Categories Positive Negative
Usability 6 4
Utility 3 2
Aesthetics 7 0
Gameplay 5 3
Challenge 6 0
Social interaction 7 5
Interest 6 0
Miscellaneous 3 0
Sum 43 14
Table2 -The categories of the reasons for general UX curve
Reason Categories Positive Negative
General UX 43 14
Ease of use 41 26
Immersion 43 10
Social interaction 38 12
Engagement 32 10
Total 197 72
Table 1- Number of reasons for general and specific UX
dimensions

12. Indicative results (3)
 All except for two “Ease of use” curves were improving or stable, with
pragmatic-related reasons to be the most frequent.
 most negative perceptions related to usability reasons
 Overall, most of the issues influencing the long-term user experience
in the game were related to non-pragmatic issues such as fun,
immersion, challenge, interest and control.

13. Key Takeaways
 It is crucial to measure UX in the long term
 Retrospective methods can greatly help in understanding how UX
changes over time
 Memories are as or more important than actual experiences
 UX Curve is a cost efficient and effective method to use for
analyzing and understanding long term UX
 The new proposed version of UX Curve (customized to gaming
environments) seems to be a valid and effective method for
assessing long term UX of MMOG players
 It can be applied to many other contexts for evaluating
products/services/systems

14. Development of an E-learning UX
Measurement System

15. Context of the study
 Based on:
 Thesis submitted by Evangelos Loutas - thesis title: “Development
of an E-learning UX Measurement System” - May 2013
 Usability and UX measurement in e-learning applications
 Studies show that usability/UX is a crucial success factor in e-
learning and facilitates learners to achieve learning objectives and
gain knowledge effectively and efficiently
 The main objective of the study:
 To develop a web-based system that will measure UX, according to
Mo2L usability evaluation method.

16. Mo2L questionnaire
 The Mo2L instrument can be used to measure different dimensions
of the e-learning user experience,
 i.e. from typical usability attributes such as content, navigation and
learnability etc. to affective learning issues such as motivation to
learn.
 A validated questionnaire with 49 items along 8 scales:
 Content, Learning design and support, Visual design, Navigation,
Accessibility, Interactivity, Self-assessment & Learnability and
Motivation to Learn

17. UX e-learning measurement
 Towards quantification of e-learning usability perceptions
 The method provides calculation of a total “e-learning usability
value” which takes into consideration all items on the questionnaire
and reflects a global usability score of the e-learning
application.
 In addition calculations are made along all the different scales of the
questionnaire presenting thus quantifiable information for all the
aspects of an e-learning application.

18. Main workflow of the system as developed
1. Interested researcher or practitioner has to register
2. A new instance of the system is created and a link to the researcher is sent.
Researcher can use this link to initiate his/her usability study by calling
users/participants.
3. The enquiry participants will be provided with a 49-item questionnaire that
contains also some demographics questions.
4. All questions are mandatory. Failure to reply to a question will result to the
invalidity of the participant's response.
5. After a certain period of time and a fair number of enquiry participants the
questionnaire link will close and further information will be given to registered
users on how to access the reports containing the usability measures.
6. The reports are extracted by using an automated procedure. Custom report
generation is possible upon request to the administrator.

19. The main features of the system
 Reports for the mean Global E-learning Usability Score (GEUS)
 Reports for all the respective Mo2L scales: mean Motivation to Learn
score, mean navigation score, etc.
 Reports for individual scores and
 Reports for open questions

20. Impact and future research
 From May 2013 – today more than 70 researchers and practitioners
around the world have expressed their interest to use the system in
their usability and UX studies.
 Quite recently five instances of the system were open and being
used by researchers from US, England and Portugal.
 Shortly we are planning a redesign of the system in order to make it
more robust and usable
 Check out the site:
 http://www.elearning-usability.com/index2.html

21. Development of a serious 3D game as a
tool for organizational learning

22. Context of the study
 Based on:
 Thesis submitted by Demetrios Mouzouros, thesis title: “Development
of a serious 3D game as a tool for organizational learning” - August
2012
 Organizational learning processes and human development
 New methods, techniques and tools are needed to effectively support
process such as onboarding, etc.
 The main objective of the study:
 To develop a serious game and use it as an organizational learning
tool
 To empirically test it for onboarding employees in a real company
environment

23. The development of the “Knowledge Donor”
 The game takes place in a 3D world that simulates a business
environment and gives the user a realistic feeling of an actual
corporate training.
 The game scenario is based on two main pillars:
 a) the existing procedure that takes place in an insurance company
for the orientation of new employees, and
 b) the fact that a percentage of the profit of this company goes to
charity (strong relations of the company with a scholarship
foundation)

24. The main game scenario
 The player is a new employee. Other employees of the company will
take the player through an orientation and initial training session
 Content about the company creation and history, the products and
marketing routines and the personnel of the company.
 The game consists of three stages.
 At the end of each stage, the player is tested for the “knowledge”
that has acquired within specific time limits
 Successful completion of the tests will reward the player with money,
all of which is collected as a donation to the scholarship foundation.

26. Empirical study
 The “Knowledge Donor” game was evaluated within a period of 20
days where the employees of the insurance company had the
chance to play the game.
 After the play sessions, interviews were conducted (by the student)
to get feedback from the players.
 Along with the interviews, the users were asked to fill-in a user-
experience questionnaire which was based on the AttrakDiff Lite
Questionnaire.
 The questionnaire measures hedonic qualities and pragmatic
qualities through 10 items that are presented in a 7-point semantic
differential scale

27. Some key findings
 Reactions and comments of the users revealed a positive approach towards
this new intervention and a great interest in playing the game.
 Almost 80% said that this could be a new and effective method to support
onboarding processes in the company.
 The same users mentioned that the game helped them refresh their
memory regarding certain things about business processes.
 However only 50% would choose this method over the traditional face-to-
face training sessions.
 Users also reported some problems with their interaction with the 3D world.
 They faced mainly navigation problems and they complained regarding
the content

28. Main limitations and future prospect
 Sample of participants
 Duration of the study
 Future implementations should be based on detailed UX driven
process of capturing requirements
 Future implementation must involve the real target users: new
employees that join the company

29. LEARNING GEOGRAPHY THROUGH SERIOUS GAMES:
THE EFFECTS OF 2-D AND 3-D GAMES ON LEARNING
EFFECTIVENESS, MOTIVATION TO LEARN AND USER
EXPERIENCE

30. Context of the study
 Based on:
 Thesis submitted by Ioanna Chatzeparaskeuaidou, thesis title:
“Development of an educational game as a tool for learning geography in
elementary schools” - August 2012
 Design of 2-dimensional and 3-dimensional serious games in education
 Limited knowledge and relevant empirical research in formal educational
settings that compares their effectiveness
 The main objective of the study:
 To develop an educational game for learning geography in elementary
schools – develop two versions a 2D and 3D
 To empirically compare their effects on learning effectiveness, motivation
to learn, and UX.

31. Empirical study
 Research question: What effects do 2D and 3Dversions of the game have
on learning effectiveness, motivation to learn and user experience?
 The game:
 Design of two versions of the game with Scratch platform and Kodu.
 The game scenario: quite simple and structured around an interactive
map of Greece.
 The game follows a question and answer method: it prompts the player to
choose one geographical region and continues with several related
questions that are region specific. This is happening repeatedly for other
regions as well and the total number is twenty questions.

32. Snapshots of the game

33. Snapshots of the game

34. Experimental process
 Participants were students from eight elementary schools located in areas of
Northern Greece.
 159 students participated in the experiments, 94 girls (59.1%) and 65
boys (40.9%), attending the 5th and 6th grade, from 16 urban and
suburban parts.
 The age ranged between 10 and 12 years old.
 Data were collected through questionnaires and mini interviews:
 a demographic survey was employed along with a multiple choice test for
assessing learning effectiveness on Geography and two scales
measuring motivation to learn and user experience

35. Experimental process
 Set up: A control group with students playing the 2D version of the game
and the experimental group with students playing the 3D version (Students
were randomly selected based on the "lottery" method)
 A mixed between-within subjects ANOVA model:
 A session (2: Pretest, Posttest) x design modality (2: 2D, 3D) ANOVA was
conducted with session as the within-subjects factor and design modality
as the between subjects factor
 The experiments took place at the computer labs in 8 elementary schools.
Participation of the students and their teachers was on voluntary basis
during the whole process

36. Experimental process and results
 Before running the mixed ANOVA an independent samples t-test for
differences between the mean scores of male and female subjects was
performed. - No statistical significant differences were found.
 Learning effectiveness:
 It was found that there is a significant interaction between design modality and
session, Wilks Lambda = .84, F (1, 149) = 27.08, p < .0005, partial eta squared = .15.
 There was a substantial main effect for session (pretest and posttest), Wilks Lambda =
.72, F (1, 149) = 57.05, p < .0005, partial eta squared = .27, with both groups showing
an increase in learning effectiveness scores across the two time periods.

37. Results
 Learning effectiveness: players in 2D group performed better in geography
knowledge tests than players in 3D group.
 Authors observed that:
 students in 2D group were more focused on the successful completion of
the game while
 students playing the 3D game were almost distracted from the visual
design of the game trying to learn to navigate and willing to explore the
elements of the environment beyond the learning focus of the game

38. Results
 Motivation to learn:
 Results revealed that there is no significant interaction between design
modality and session, Wilks Lambda = .97, F (1, 149) = 3.19, p = .076,
partial eta squared = .021.
 There was a substantial main effect for session (pretest and posttest), Wilks
Lambda = .81, F (1, 149) = 33.69, p < .0005, partial eta squared = .184, with
both groups showing a decrease in motivation to learn scores across the
two time periods

39. Results
 Motivation to learn:
 We observed that students entered the experimental process with a great
enthusiasm
 while they were engaged with game activities they oriented themselves
more towards fun than learning.
 This may explain the fact that motivation to learn decreased after the end
of the experiment. Confirmed in other studies as well (Yang 2012, Annetta
et al. 2009), as the novelty effect fades motivation gets lower.

40. Results
 Motivation to learn:
 The main effect comparing the two versions of game was also significant,
F (1, 149) =7.71, p = .006, partial eta squared = .049, suggesting a
difference between the two design modalities.
 We see that players in 3D group showed greater motivation to learn than
those who played the 2D version.

41. Results
 UX:
 There is no significant interaction between design modality and session,
Wilks Lambda = .99, F (1, 149) = .28, p = .595, partial eta squared = .002.
 There was a substantial main effect for session (pretest and posttest),
Wilks Lambda = .77, F (1, 149) = 42.46, p < .0005, partial eta squared =
.222, with both groups showing a decrease in user experience scores
across the two sessions
 Observation: Students had great expectations in terms of anticipated UX in
both groups

42. Results
 UX:
 The main effect comparing the two versions of game was also significant,
F (1, 149) =4.75, p = .031, partial eta squared = .031, suggesting a
difference between the two design modalities.
 players in 3D group expressed a higher degree of user experience than
the players in the 2D group.
 We argue that 3D learning environments are supposed to enhance the user
experience in terms of flow, presence, etc.
 We attribute this effect on the perceived hedonic qualities of 3D
environments such as novelty and greater fidelity.

43. Future research
 Longer experiments are needed
 Different types of knowledge tests
 Design modalities (2d and 3d) can be combined in a single game
 The games could be designed in a multiplayer mode with a different
pedagogical focus where activities will require collaboration for successful
completion of the game.

44. Some papers associated with the theses
 [C21] Chatzeparaskeuaidou, I. and Zaharias, P. Hedonic and pragmatic qualities as
predictors for motivation to learn in serious educational games. Accepted at the
Foundations of Digital Games 2013, Workshop on Games for Learning.
 [J17] Chatzeparaskeuaidou, I. and Zaharias, P. The effects of a 2D and 3D game on
learning effectiveness and motivation. Submitted to Journal of Educational
Technology & Society
 [C24] Moshou, E. and Zaharias, P. (2013). The UX Curve revisited: Assessing long
term UX for games. Accepted at the Workshop on Designing Gamification: Creating
Gameful and Playful Experiences at CHI 2013 - Conference on Human Factors in
Computing Systems.
 [J19] Zaharias, P. and Moschou, E. Drawing Curves for assessing long term UX of
Massive Multiplayer Online Games. Submitted to Interacting with Computers
 [J16] Mouzouros, D. and Zaharias, P. Development of a 3D serious game for
knowledge management: an empirical investigation. Submitted to Journal of Virtual
Worlds Research

45. Thank you very much !
Any Questions?
Contact info:
 panagiotis.zacharias@ouc.ac.cy | pz@aueb.gr
 http://about.me/panagiotis_zaharias
 http://gr.linkedin.com/pub/panagiotis-zaharias/0/bb6/a91