Evaluating the User Experience

1. Evaluating User Experience
MGA 673 – UX evaluation
Sónia Sousa
Tallinn University
School of Digital Technologies

2. Why we need
to evaluate?

3. Evaluation
• Main idea
• Get informed about the value of your product:
• idea, conception, users’ needs, understandability, learnability,
trust...
• Getting a feedback about your design issues:
• Addressing end users and collecting data about their
perception and usage patterns.
• To better understand users experiences,
• Letting them to reflect on their needs and experiences
• meanwhile trying to make sense about the context, stimuli and
time factors
Why we
need to
evaluate?

4. Purpose
Evalu-
ation
Artefact
Benefici-
aries
Interaction design

5. When?

6. Evaluation (When?)
Development phase
– Scenarios, sketches; i.e.
concepts
– Early prototypes
– Functional prototypes
– Products on market

7. Where?

8. Evaluation (Where?)
Study type:
– Field studies
– Lab studies
– Online studies
– Questionnaires / Scales

9. Who to use?

10. Evaluation (Who to use?)
Participant
– UX experts
– One user at a time
– Groups of users
– Pairs of users

11. What to
evaluate?

12. Evaluation (What to evaluate?)
Study period:
– Before usage
– Snapshots during interaction
– An experience (of a task or activity)
– Long-term UX

13. What Objects
evaluate?

14. Evaluation (What to evaluate?)
Interactive objects
– Interface on a screen
– Platform
– Browser
– Mobile
– Spatial interactive objects
Information exchange
• Visual
• Audible
• Tangible

15. Evaluation
• Let’s then take a look into concepts such as
• Evaluation Concepts: usefulness, usability and user
experience
• Evaluation Methods and Tools
– Objects for Evaluation
– Methods & categories & metrics
– Data & Scales
2016 Sónia Sousa 15

16. Evaluation Concepts:
usefulness, usability and user experience

17. a good user experience depends
on both pragmatic and hedonic
qualities

18. Usability
Interaction design
User
experience
Useful concepts
Pragmatic
qualities Hedonic
qualities

19. usefulness and usability are
pragmatic

20. pleasure is hedonic

21. pleasure is hedonic

22. Usability
Interaction design
Useful concepts
Pragmatic
qualities
the extent to which a product or service can be
used by specific users to achieve predefined goals
in a specified context of use
How? Assessing products or service
effectiveness, efficiency and satisfaction.
Consider also learnability and memorability.
Usefulness
the degree to which a product enables a user to
achieve his or her goals,
How? assessing user’s willingness to use the
product or service.

23. Interaction design
User
experience
Useful concepts
Hedonic
qualities
A person’s perception and responses of use or
anticipated use of a product or service.
How? Assessing additionally experiences
emotions, beliefs, preferences, perceptions,
physical and psychological responses, behaviors
and accomplishments that occur before, during
and after use.
User perception
the degree to which a user perceives the product
or service as providing a good experience,
How? assessing user’s intentions to use the
product or service.

24. Usability and user experience
2016 Sónia Sousa 24

25. How to…choose
a specific method?

26. evaluating user experiences is
tricky but possible

27. and we will now address possible
approaches

28. Methods and Tools
User Experience and Research

29. Methods & categories
• Categories help to pick suitable method
according to study conditions.
2016 Sónia Sousa 29
Methods
• Formative
• Summative
• Design study
• Experimental study
• Comparative study
Categories / study Type
• Field studies
• Lab studies
• Online studies
• Questionnaires / Scales

30. Methods & categories
Categories / Time factor
• Before usage
• Snapshots during
interaction
• An experience episode (of a
task or activity)
• Long-term UX
Method / Measures
• Self reported
• Observed
• Directly measured
(objective)
• Latent value (calculated,
concluded)
– The "Big Five personality
traits" have been inferred
using factor analysis.
• Expert analysis (opinion)

31. Data & scales
Metrics / Data
• Time to complete task
• Count of errors and/or
effort
• Written report
• Schemas/diagrams
• Audio recordings
• Video recordings
• Images/photos
• …
Metrics / scales
• Indicators
– Effectiveness
– Efficiency
– Satisfaction
• Likert scale
• Comparison values
• Visual values
• Emotional values

32. Other conditions
Other conditions
• Conditions of use
• Surrounding environment
• Human factor
– Prejudice
– Previous experience
• User profile
• Demographic background
• Emotional background
• Experience level
Outcome of UX study
• What and How certain
behaviours occurs
• UX value
• Explaining the issues
– Why it occurs?
– How to solve them?
2016 Sónia Sousa 32

33. Most known Methods…
• Usability Methods
• System Usability scale [SUS]
• Methods that were transposed from Usability
• Cognitive Walkthrough
• Think aloud
• Diaries
2014 Sónia Sousa & Mati Mottus 33

34. Most known Methods…
• Others are adaptation of
– Extended usability testing
– TRUE Tracking Realtime User Experience
– UX Expert evaluation
– Property checklists
• Others are new
– AttrakDiff -
– Emocards, Emotion Cards, Emofaces
– Reaction Cards
2016 Sónia Sousa 34

35. SUS is a quick and dirty scale
popular for end-of-test
subjective assessments

36. System Usability Scale

37. for positively-worded items [1, 3, 5, 7, 9]
the score contribution is the scale
position minus 1
for negatively-worded items [2, 4, 6, 8,
10] it is 5 minus the scale position
to get the overall SUS score, multiply
the sum of the scores by 2.5

38. SUS scores range from 0 (worst)
to 100 (better)

40. AttrackDiff

41. AttrackDiff assesses the user's
feelings about a system with a
questionnaire addressing both
hedonic and pragmatic
dimensions of UX

42. AttrackDiff explores…
pragmatic qualities
identity
stimulation
attractiveness

43. technical...human
complicated...simple
impractical...practical
cumbersome...straightforward
unpredictable...predictable
confusing...clearly structured
unruly...manageable

45. http://attrakdiff.de/

46. Inspection methods
• There are also several but we will address:
– Cognitive walk-through
– Heuristic evaluation

47. Testing methods
• There are several but we will address:
– Co-discovery
– Wizard of Oz

48. Benefits
• This technique can be used
to test device concepts and
techniques and suggested
functionality before it is
implemented
48Image source: http://stanford.edu

49. Testing methods
• This kind of evaluation should be repeated
until no significant added value comes from
bring in an additional subject
– The usual number is 5, but other claim otherwise

50. Pre and post-tests
• When applying testing methods, pre-test and
post-test questionnaires are also used to
gather feedback on the product being tested
– A common questionnaire is the 25 years old
System Usability Scale

51. Evaluation protocol
• But these methods are useless without an
adequate evaluation protocol
• While designing your evaluation protocol, you
should take into account…
– A mixed selection of inspection and testing
methods;
– The user stories supported by your prototype; and
– The affordances of your paper prototype.

52. What we discussed
• Why we need to evaluate?
• Main idea, purpose of evaluation
• When and, Where we need to evaluate.
• What to focus during the evaluation process?
• Who shall we use?
• The object of evaluation
• Evaluation Concepts
• Usefulness, Usability and User experience
• Methods and Tools
• How to choose a specific method of evaluation?

53. Future aims
• Focus on Methods and tools
• Design a evaluation procedure
• Implementation protocols
• How to Apply the most known methods
• During the different development phases
• Concept
• Low fidelity
• High fidelity

54. In sum…
• As mentioned before
• In general, product and service evaluation can be
formative or summative
• Low-Fi prototype evaluation is mostly formative
• Not so rigorous procedure of implementation
• Hi-Fi evaluation
• Include both formative or summative
• The method for data collection should be mixed
• The evaluation procedure should be carefully planed
• The quality of results depend on it