Usability testing for Voice applications Abi Reynolds Steven Fullerton

1. Voice recognition in Scotland?
1Source :https://giphy.com

2. Designing Voice application and testing
with Wizard of Oz
Abi Reynolds
Steven Fullerton
June 2019
2

3. 3
Abi Reynolds
Principal UX consultant
@AbiJReynolds
Steven Fullerton
UX consultant
@StevenFullerton
Who are we?

4. Why are you interested in Voice?
4

5. Our Role in Conversation Design today.
5Image source: unsplash photo @bradneathery

6. Good and bad implementation
6

7. Good and bad implementation
7

8. Why Voice?
8

9. Why Voice?
9

10. What do we mean by voice?
Google Home Amazon Echo
10Image sources: unsplash photos @andresurena and @@benceboros

11. Designing conversations
11
Keeping the user at the centre
What could a user centred design process for Voice look like?

12. Discovery
Requirements=
Intents
Script=
sample
dialogues
Prototype =
dialogue
flows
Test =WOZ
test/iterate
Launch
12
Research
DesignTest

13. Intents = Purpose
13Source :https://giphy.com

14. What does a smart speaker hear?
14
Alexa, Ask Tile to find my Keys

15. What does a smart speaker hear?
15
Alexa, Ask Tile to find my Keys
Utterance

16. What does a smart speaker hear?
16
Alexa, Ask Tile to find my Keys
Wake Word
Utterance

17. What does a smart speaker hear?
17
Alexa, Ask Tile to find my Keys
Wake Word Invocation
Utterance

18. What does a smart speaker hear?
18
Alexa, Ask Tile to find my Keys
Wake Word Invocation Intent Slot
Utterance

19. 19
“Design for how people talk, rather than how
you wish them to talk.”
- Cathy Pearl

20. Amy the Hotel Concierge
▪ Our business objective: To cut down
on calls to reception (save costs).
▪ Our customer experience objective:
We think Amy will provide an
innovative and useful service that will
add to the overall the guest
experience.
A voice app should be easy to use hands
free within hotel room whilst guests are
doing other things.
20

21. Activity – 10 Minutes
Start the conversation!
Write down:
• A two way conversation between a
customer who wants to know when
checkout is and the smart assistant
concierge.
• The concierge can upsell late checkout
at a cost of £10, IF the customer
wants it.
Create a short script based on the
scenario above and work in pairs or
small groups…
21
Activity – 10 Minutes
Start the conversation!
Write down:
• A two way conversation between a
customer who wants to know when
checkout is and the smart assistant
concierge.
• The concierge can upsell late checkout
at a cost of £10, IF the customer
wants it.
Create a short script based on the
scenario above and work in pairs or
small groups…
22

22. Feedback
How did it go?
How many different utterances did we get?
Lets act out your script
How could the flow be improved or made to feel more natural?
22

23. Discovery
Requirements=
Intents
Script=
sample
dialogues
Prototype =
dialogue
flows
Test =WOZ
test/iterate
Launch
23

24. 24Source NNG Wireflow https://www.nngroup.com/articles/wireflows/

25. Dialogue flows gives you an overview of all
directions the conversations can go (based
on intents). It plots how you plan to
respond to questions and guide the user
towards their goal.
25

26. 26
https://dialogflow.com

27. Activity – 10 Minutes
Complete the extended check-out flow
The hotel offers extended check out at 3pm for an
extra 10 pounds, which can be added to the room
bill
Work in groups and using the sheets provided
1. Complete the steps in the path until the user reaches
their goal
27

28. 28

29. Remember
1. Core conversation - Y/N proceed. Until end of task
2. Prompt to go forward or finish task
3. Think how this could go wrong and how Amy could recover the conversation
29

30. Feedback
How did it go?
How many steps did you come up with?
What error messages did you come up with…
Here is one we did earlier…
30

31. Think about… (when starting to write conversations..)
▪ Ask clarifying questions at start
▪ Make assumptions and build for there, make clarifications throughout
▪ Think about how to present choices (and cater for human’s limited working memory)
▪ Avoid overloading the skill with instructions
▪ Think personality and tone and vary responses
▪ Keep the conversation moving forward and bring user closer to their goal
▪ Consider how to repair flow if user goes off happy path – how to get user back on
track or fail gracefully.
31

32. 32Source Google https://developers.google.com/actions/downloads/conversational-repair.pdf

33. WOZ Testing
Testing voice app using WOZ Testing
33

34. Wizard of Oz (WOz) Approach
In the WOz test, participants interact with a VUI such as
Alexa, whom they believe to be ‘live’, but which is actually
being operated by unseen researcher in an another room.

35. what the user sees the wizard
Wizard of Oz in Voice usability testing

36. WOZ Testing
1. Understand main use cases/tasks - user intents
2. Construct sample dialogue flows and responses for each task
3. Create a moderator script including scenarios and metrics for gathering
feedback
4. Create audio files for each flow include messages for error recovery etc. (can
use Amazon’s Polly or prototype tool such as Sayspring)
5. Connect audio files soundboard (can use soundboard lite)
6. Set up WOZ test
7. Record session (can use audacity, Evernote etc.)
36

37. Running a Voice usability session
37
Research brief Test stimulus Test
plan/moderator
script
Set up and record
the session

38. Research brief
38

39. Test Stimulus
39
1 2 3

40. 40
Welcome to UX Scotland
Image source: Unsplash photo @ugsdreamer
Polly system
Speech into text from Amazon
Web Services

41. Moderator script
41
Very
difficult
Difficult
Slightly
difficult
Neutral Slightly easy Easy
Very
easy
1 2 3 4 5 6 7

42. Run and record session
42
Amy
How can I help?

43. What did you think?
Tasks/Flows?
Language used?
Amy’s Personality?
Anything else?
43

44. 44
https//:tortu.io
Prototyping tools
https://dialogflow.com
https://www. sayspring.com
https://botmock.com

45. Finally… Some tips when testing Voice apps
▪ Test to see if your conversational flows match user intents
▪ It’s important not to mention command words or ask leading questions in TEST as we want to find our more
what language and synonyms participants use in testing
▪ Check how your app handles feedback, contextual help and errors
▪ Collect feedback on Persona of app
▪ Use metrics such as ease of use and satisfaction to measure user experience
▪ Agile methodology and quick to iterate.
*Remember it may be agile and quick to do but its important to follow a
robust research process and have clear research aims and objectives
45

46. @UserVision
www.uservision.co.uk
Abi Reynolds
@Abijreynolds
abi@uservision.co.uk
hello@uservision.co.uk
Steven Fullerton
@StevenFullerton
Steven.fullerton@uservision.co.uk