1. Evaluating the User Experience
Sónia Sousa & Mati Mottus
2014 0

2. Product evaluation
2014 Sónia Sousa & Mati Mottus 1
• Nowadays we need to more than just
– Efficient, effective and cheap products.
• We need products that stimulate pleasure;
and
– Address our needs for social and individual
differentiation.

3. Usability Vs UX evaluation
Usability is concerned
• with product attributes and
prevent errors
UX is concerned
• in build positive experiences
resulted from the
interaction with the product
2014 Sónia Sousa & Mati Mottus 2
using
Task
oriented
Perceived
usage
Prevent Experience Reflection

4. Project 
• The aim is
– To ” Evaluate HOW user’s perceive a certain product“
• This should be done by comparing different
features of the product
– What would be the difference between
• using a desktop version and the mobile version..
• The product needs to be:
– Ubiquitous,
• should include a Desktop, mobile and or tablet version.
2014 Sónia Sousa & Mati Mottus 3

5. This project contain 3 main activities
• Present your project planning procedure,
– due date: 24.04
• Project implementation,
– until 08.05
• Present the results,
– due date: 09.05
• Deliver the Report,
– due date: 26.05
2014 Sónia Sousa & Mati Mottus 4

6. The project planning,
• Should include the following issues,
– Forming the working groups;
– Assemble the group environment;
– Define your UX evaluation target and Study it carefully;
– Plan your evaluation procedure;
– Identify the participants;
– Choosing the methods;
– Plan on How to Implement the study;
– Write in detail the evaluation procedure;
– Pilot study;
– Implement and analyze the results; and
– Write the report.
2014 Sónia Sousa & Mati Mottus 5

7. Planning the project
due date: 24.04
2014 Sónia Sousa & Mati Mottus 6

8. Step 1
• Group work organization
– Forming the working groups
• Maximum number of group members is 3
• Assemble the group environment,
– Define each member shared responsibilities and
roles
– Assemble the various tools and services you need.
2014 Sónia Sousa & Mati Mottus 7

9. Step 2
• Define your UX evaluation target
– Study it carefully
• Write in a paper a description of the product you will
evaluate.
• To do it for example
– you can turn to your colleague and try (in your
own words) to
• Describe what your product represents to you in 5
seconds.
2014 Sónia Sousa & Mati Mottus 8

10. Step 3
• Plan your evaluation procedure
– Answer to the questions:
– What you need to observe
• tasks, content, layout, and so on
– that enables you (as researcher) to get
• information on HOW user's perceive your product.
– Write the main ideas on a paper.
2014 Sónia Sousa & Mati Mottus 9

11. Step 4
• Identify the participants
– Clearly identify who is going to participate in your
study. Focus on describing
• Age, gender, technical knowledge, social status and so
on
• Write the main ideas on a paper.
2014 Sónia Sousa & Mati Mottus 10

12. Step 5
• Choosing the methods
– Think on…
• What methods I am going to use and When?
• How will I do it?
• Answer to the questions
– What is my strategy to achieve my objectives?
– What is the order of my actions?
– How will I use different resources in my actions?
• Write the main ideas on a paper.
2014 Sónia Sousa & Mati Mottus 11

13. Step 5
• Plan on How to Implement the study
– Step 1:
• Start by developing a measurable criteria to evaluate
your planned activities.
– Reflect on what tools to use; and how to design
those tools.
2014 Sónia Sousa & Mati Mottus 12

14. Step 6
• Write in detail the evaluation procedure
– Use schemas and diagrams to explain the
procedure
– Use bullet list as well
2014 Sónia Sousa & Mati Mottus 13

15. Pilot-study
• Pilot study
– After designing the tools "Do a pilot-test" and
change what is needed to change.
• Ask yourself...
– before present your ideas to the group:
– How do I know that I will be successful?
– How will I use the different resources?
2014 Sónia Sousa & Mati Mottus 14

16. Reflection session:
• Write down your project working plan and
procedure
• Present to the group
– please use
• the Gdocs presentation provided by the teacher.
• And the excel file to write your project working plan
2014 Sónia Sousa & Mati Mottus 15

17. Systematic measure
?
2014 Sónia Sousa & Mati Mottus 16

18. Category of UX methods
• Study types
– Field studies
– Lab studies
– Online studies
– Questionnaires / Scales
• Studied period of
experience
– Before usage
– Snapshots
– An episode (of a task or
activity)
– Long-term UX
• Development phase
– Concepts
– Early prototypes
– Functional prototypes
– Products on market
• Evaluator / Information
provider
– UX experts
– One user at a time
– Groups of users
– Pairs of users
Sónia Sousa & Mati Mottus, TLU, 2014
Source: allaboutux.org

19. UX Methods…
• Some Methods are transposed from Usability like…
– Cognitive Walkthrough
– Think aloud
– Diaries
• Others are adaptation of…
– Extended usability testing
– TRUE Tracking Realtime User Experience
– UX Expert evaluation
– Property checklists
• Others are new…
– AttrakDiff, Emocards, Emotion Cards, Emofaces
– Reaction Cards
2014 Sónia Sousa & Mati Mottus 18

20. How to evaluate?
2014 Sónia Sousa & Mati Mottus 19
Utility
Usability Pleasure
Intentions Social Value
Efficiency Perceptions of
interactions
The product
Affection
Satifaction
Social Value
Mood State
Relevance of the product
User intention to use
usefulness Trust
Learnability
Accessibility
UX
Asthetics
Social Links

21. When to evaluate?
2014 Sónia Sousa & Mati Mottus 20
EmotionsValue Perceptions
User AttitudesProduct Qualities Usage IntentionsUser Beliefs
Before interaction During interaction After use
+
First impressions Mood State
Why people like and use certain products What make people use a certain product

22. What to evaluate?
2014 Sónia Sousa & Mati Mottus 21
Product
Layout, utility, functionalityFirst impressions
Hedonic qualities
Emotions
Mood state
Value
Perceptions
Pleasure, Affective arousal
symbolic aspects of the product
User perceptions

23. How to collect the data
2014 Sónia Sousa & Mati Mottus 22
What is the usage intention?
Positive outcomes
Motivation aspects
Granularity measure methods
Value Vs importance of use
Low level experience measure
Overall experience measure
+ +
AttitudesIntentions
What is the Usage attitude
User eXperience

24. First impressions
• Aesthetics and interaction features
– Layout, content, functionality
• Study period
– before and during use
– Personal expectation and needs
• Record of mouse activity and mood state
• Possible methods
– Visual and verbal tests
2014 Sónia Sousa & Mati Mottus 23

25. Value & perceptions
• What is the intention to use?
– Judgments of the product based on
• Feelings towards the product
• Hedonic qualities and symbolic aspects of the product
– Pleasure, challenge, needs, expressions
• Measurements
– AttractDiff
– EMG values
– Efficiency
– Effectiveness
– sensors
2014 Sónia Sousa & Mati Mottus 24

26. Emotions & Mood state
• Pleasure and affective arousal
– Positive outcomes
• Emotions, joy, excitement
• Measurements
– Affects
– BMIS
– MIS
– RAS
– Affect grid
– SAM
– PreEmo
2014 Sónia Sousa & Mati Mottus 25

27. 2014 Sónia Sousa & Mati Mottus 26
Introduction
start the
procedure
task 1 to 3 Usability
record
mouse
activity
video
record
task 1
First
impressions
Question...
tasks 2 to 3
Values
perceptions
EMG
BEFORE
DURING
BEFORE DURING/
tasks 4 Emotions EmoCards
AFTER
End the
procedure
Consent
form
Acessibility Heuristics

28. Project report
• Report
– Eliciting the positive aspects of users experience
• Emotions, feelings and motivation
– Pragmatic qualities, hedonic qualities and
emotions
• Utility – perceptions of need
• Satisfaction – mood sate
• Efficiency
• Accessibility
2014 Sónia Sousa & Mati Mottus 27