An update to my Introduction to Games User Research lecture (http://www.slideshare.net/Gortag/an-introduction-to-games-user-research-methods). Due to a changing course design this version focuses a bit more on questionnaire design and interviews. A few other changes have been made and the aesthetics have also been changed.

1. Games User Research
b.lewis.evans@rug.nl
Ben Lewis-Evans

2. b.lewis.evans@rug.nl
@ikbenben

3. XBL: Gortag
PSN: LaglGortag
Steam: laglgortag
b.lewis.evans@rug.nl
@ikbenben

4. Today
 What is Games User Research?
 User Research & Evaluation
Methodology

5. QA
 Testing the game
(bugs, technology,
etc)

6. GUR
 Examining the
player
 Fun & user
experiences

7. Fun
 What is fun?
 Easy to use
 Challenging
 Emotional impact
 Engaging
 Compelling
 Relaxing
 It is subjective!

8. Sometimes fun High Activation Often fun
Unpleasant Pleasant
Low Activation
Almost never fun

9. General tips
 Get (enough)
representative users
 The game is being
tested, NOT the user
 Work out what you
want to know

10. General tips
 Test as early as
possible, fix problems,
& then test again
(RITE)
 Listen to problems,
not necessarily
solutions
 Not (primarily) for
balance & bugs, but
for fun!
 Impact evaluation is
different from general
GUR

11. http://www.polygon.com/2012/10/24/3538296
/data-entry-risk-management-and-tacos-
inside-halo-4s-playtest-labs

12. Research Methods

13. Focus groups
 6-10 people
 Lead by a facilitator
 Specific questions
 Try the
game/discuss
potential ideas
 Talk about it

14. Focus Groups
 Pros
 More people can = more feedback
 Gets everyone together in one place
 Follow up questions
 Good for discussing concepts
 Cons
 You need a good facilitator
 Strong voices may take over
 Too many “helpful” suggestions
 What people say is not often
what they do
 Not for evaluation of impact

15. Heuristic Evaluation
 Expert evaluation (very loosely like
a game review)
 Are clear goals provided?
 Are players rewards meaningful?
 Does the player feel in control?  Is the game and the outcome
 Is the game balanced? fair?
 Is the first playthrough and first  Is the game replayable?
impression good?  Is the AI
 Is there a good story? visible, consistent, yet
 Does the game continue to somewhat unpredictable?
progress well?  Is the game too frustrating?
 Is the game consistent and  Is the learning curve too steep
responsive? or too long?
 Is it clear why a player failed?  Emotional impact?
 Are their variable difficulty  Not too much boring
levels? repetition?
 Can players recognise
important elements on screen?
(Christina et al 2009)

16. Heuristic Evaluation
 Pros
 Smaller numbers
 Experts are experts  Cons
 You need experts
 Which heuristics
to pick?
 Experts are
experts

17. Questionnaires &
surveys
 During gameplay
(at or after set
moments)
 After gameplay
 Ask for what
interests you
 Allow for some
open ended
answers

18. I can’t show the whole scale here. If you want the GEQ go to
http://www.gamexplab.nl/
Game Experience Questionnaire (GEQ)
http://www.gamexplab.nl/

19. The MANIKIN scale

20. The Affect Grid

21. Custom Questionnaire

22. How to design a
questionnaire
1. Work out what you
want to know
2. Design the
questions/statements
3. Create the
questionnaire
4. Test the questionnaire

23. 1. Work out what you
want to know
 Brainstorm, make lists,
look at your design
documents
 Then cut it down
 Only what you need to
know
 Aim for no more than
15 minutes

24. 2. Design the questions/statements
 Questions vs Statements
Questions are good for gaining
information (age, ratings of fun,
etc)
How much fun did you have in
the last quest?
1–2–3–4–5-6-7
None A lot

25. 2. Design the questions/statements
 Questions vs Statements
Statements are good for
assessing attitudes
i.e. agreement with an idea
The last quest was a lot of fun
1–2–3–4–5-6-7
Strongly Strongly
agree disagree

26. 2. Design the questions/statements
 Use clear, everyday language
I really felt like I identified with the
motivations of the main protagonist in the
game
The main character in the game was just
like me

27. Can players identify
these enemies by
name?
Consider visual aids

28. Without images would your players
know a M4A1 from a G36C?

29. 2. Design the questions/statements
 Closed or Open?
Indicate your agreement with the following
statement
This is one of the best research methods
lectures I have ever attended:
1 2 3 4 5
Strongly – Agree – Neutral – Disagree – Strongly
Agree Disagree

30. 2. Design the questions/statements
 Closed or Open?
What was your favourite weapon, and why?
_________________________________________________________
_________________________________________________________
_________________________________________________________
_________________________________________________________
_________________________________________________________
_________________________________________________________
_________________________________________________________
_________________________________________________________

31.  Yes/No, Male/Female, True/False
 Direct & precise, not very data rich
2. Design the questions/statements
 Scale:
 Dichotomous

32.  Place a mark along a continuum
 Data rich, a bit vague
2. Design the questions/statements
 Scale:
 Continuous

34.  Interval: 1 – 2 – 3 – 4 – 5, etc
 Good data, quite specific
2. Design the questions/statements
 Scale:
 Interval

35. 2. Design the questions/statements
 Scale
 Be consistent
 Problem with lazy responding?
 Use a trick question/statement
e.g. “Agree with this statement”
- Use even spacing
1 2 3 4 5
1 2 3 4 5

36. 2. Design the questions/statements
 Both the question/statement and the scale
should be clear, and exhaustive
What education do you have?
Primary School
Secondary School
Bachelors
Masters

37. 2. Design the questions/statements
What is your highest completed level of
education?
Primary School
Secondary School
Trade School
Bachelors
Masters
PhD
Other:______________

38. 2. Design the questions/statements
 Free vs Forced choice
What game types do you enjoy (select as
many as you like)
vs
What is your favourite game type (select
one)

39. 2. Design the
questions/statements
 Clear, and exhaustive
 Only a single concept/question
 Not leading
 “This game was fun, how fun
was it?”
 Not loaded
 “People with bad co-
ordination typically do badly
in our game, how did you do?”

40. 3. Create the questionnaire
 Paper vs digital
 Question order
 Number the questions
 Easy questions at the start
 Don’t lead/cue/give away
later answers

41. Qualtrics survey system
http://www.qualtrics.com/

42. 4. Test the questionnaire
 Give it to a few people to
fill in
 No plan survives first
contact with the
participants
 Revise

44.  Yes/No, Male/Female, True/False
 Direct & precise, not very data rich
2. Design the questions/statements
 Scale:
 Dichotomous

46. (sometimes?)
Neutral/Neither?

47. Never Seldom Sometimes Often Always

48. Disagree Agree

49.  Questionnaires & Surveys
 Pros
 Consistent
 Quantifiable
 Fast
 Cons
 Lack follow up
 Not objective
 Need a large(ish) sample

50. Interviews
1. Setup
2. During an interview
3. After the interview

51. Conducting an Interview
1. Setup
 Chose a good setting
 Comfortable with few distractions
 Only one or two people
 Explain why you are interviewing them
 Again, about the game, not them
 Tell them how long it will take
 Get permission to record the interview

52. Conducting an Interview
1. Setup
 Have some set questions
 Ask follow up questions
 Don’t be confrontational

53. Conducting an Interview
1. Setup
 The same rules as for in a
questionnaires
 Not leading, neutral, not loaded,
clear, only one meaning
 Avoid questions that can be
answered with yes or no
 Start with easy questions

54. Conducting an Interview
2. During the interview
 Only one question at a time
 Encourage responding
 Head nodding, “ah huh”
 But don’t let them go off on a
tangent for too long

55. Conducting an Interview
2. During the interview
 Be as neutral as possible
 “Act as if you have heard it all
before”
 Careful with note taking
 People are suspicious of
interviewers
 End with an opportunity for
interviewees to add anything
they would like

56. Conducting an Interview
3. After the interview
 If you have time make a
transcript
 Look for common threads in
what people say
 Group into themes
 May point issues you may have
never expected

57.  Interviews
 Pros
 Rich data
 Can follow up
 Good for scope
 Cons
 Less quantifiable
 Time consuming
 Not objective

58. Observation studies
 Watch/Record
 Either with a
facilitator or without
 Facilitator must be
as hands off as
possible
 Watch faces/body for
emotion
 Only write down what
you actually see!

59. >Test scripts
 Like a recipe for a
delicious cake, these lay
out how the test will go
 Order of events
 What the tester should
do and say, and when
 Clear and precise
 Could anyone follow
the instructions and
carry out a test?
 Ensures consistency!

61. Contextual Inquiry (field
studies)
 Similar to formal
observational studies
 In the wild
 Observing real use in
the real environment
 Better once a game is
released
 Or similar games
 Useful for evaluation

62. Think out loud
Observe gameplay, and
note down what they say &
when they say it
Don’t prompt them, and
don’t correct them

63. YOU DON’T COME IN THE BOX WITH THE
GAME

65. Observation studies
 Pros
 Objective data
 i.e. You see what players actually DO,
not what they say they would do
 Facilitator can help if really needed
 Cons
 Time consuming to analyse video
 Training required to get the best out of
observation (especially for emotion)
 Avoid Observer Bias

66. Gameplay Metrics
 Observation via
game data
 Number of
incidents
 Where, when,
and how they
occurred and
with who or
what?

67. http://engineroom.ubi.com/game-telemetry-
with-playtest-dna-on-assassins-creed/

68. http://engineroom.ubi.com/game-telemetry-
with-playtest-dna-on-assassin%E2%80%99s-
creed-part-2/

69. http://engineroom.ubi.com/game-telemetry-
with-playtest-dna-on-assassins-creed-part-3/

70. Gameplay Metrics
 Pros
 Objective data
 Good for evaluation
 See trends
 Cons
 Time consuming
 No subjective
feedback/context
 Needs larger sample
sizes
 Data overload

71. Biometrics/psychophysiology
 Measuring body signals:
 From the Brain (EEG), the
Heart (EKG), the muscles
(EMG), the eyes
(eyetracking), the skin
(EDA), etc
 The body gives clues into
cognition, and emotion

72. Biometrics/psychophysiology
 Pros
 Gives objective quantifiable
data
 Allows for continuous data
recording
 Cons
 Invasive
 Costs a lot of time & money to
use & analyse
 Problems with specificity,
artefacts, inference and validity

73. Summary
 Many options
 In most cases I recommend:
 Observation
 Objective data & insights
 After play (between level)
questionnaires
 Specific questions & subjective
feedback
 WARNING: People don’t often do
what they say
 Metrics are great for evaluation,
tweaking, issues, & insights

74. Summary
 Don’t wait until the game is
almost finished for GUR
 It is easier to change &
plan things early in the
process
 Listen to what people say
is wrong/right, don’t worry
too much about what they
suggest to do to fix it
 You are the game
designer

75. Credits
 All the copyright holders of the
images I have used
 A rough primer to user research
(parts 1 & 2)
 http://www.gamasutra.com/view/fe
ature/169069/
 http://www.gamasutra.com/view/fe
ature/170332/
 Christina et al 2009
 http://mi-lab.org/wp-
content/blogs.dir/1/files/publicatio
ns/uxInGames_Koeffel_et_al.pdf
 Special thanks to the GUR-SIG for
their feedback and inspiration

76. Questions?
XBL: Gortag
PSN: LaglGortag
Steam: laglgortag
b.lewis.evans@rug.nl
@ikbenben