2. #users1st
Objectives
By the end of this workshop you will be able to…
get buy-in for remote testing
design your own tests
launch them
analyse them
and present results internally.
3. #users1st
Agenda
What is remote UX Testing? [10 m]
Test Design and Launch [40 m]
Getting buy-in [20 m]
Live analysis [20 m]
Presenting results [20 m]
Q&A [10 m]
4. #users1st
The company
Started 6 years ago
N. London office
20 Full-time staff
30+ remote analysts
10,000+ panelists
1,500 + clients
250,000+ user videos
11. #users1st
2. Let’s design and launch a live test
What’s involved in designing a test?
5 x task writing tips
2 example tasks
Each table writes tasks for one of your sites
And launches it
12. #users1st
What’s involved in designing a test?
Setting objectives
Deciding on what type of test to run
Participants: Profile, Volume and Devices
Tasks for users to follow
Exit Question
1st
13. #users1st
5 x Task writing tips
1. Number each task
2. Set a scenario
3. Focus on tasks NOT opinions
4. You need not start at your Home Page
5. Aim for 20 minutes(ish)
6. Use fake personal data, stop at the point of payment
7. Run a pilot to check your tasks
14. #users1st
Start the screen recorder and click on the displayed link. (Please make sure you
follow every instruction and talk through your experience. Remember, we are
testing the site, not you.)
Thank you – you can stop the screen recorder.
At the beginning of each task list:
At the end of each task list:
15. #users1st
1. Pretend you’re looking to re-mortgage
2. Without thinking too much, say what three words come to mind about [the brand]?
3. Would you give your personal details if requested by this site? (You will not have to during
this test, but we are interested if you would do so in real life.)
4. Imagine that you wanted to apply for a mortgage to buy a new home. Where would you click
first on this page?
5. Explore the mortgage offers, as if you would be genuinely interested in applying for a
mortgage with us. (Spend no more then 5 minutes on this task.)
6. Apply for a mortgage, and proceed as far as you can. (You can use fake data, if you are not
comfortable with providing your personal details.)
7. If you had a magic wand how would you improve our website?
16. #users1st
1. Pretend you are moving to Reading and for your new home will
need internet, TV and phone service.
2. Using any and as many websites as you like find a plan (or plans)
that genuinely suit your needs.
3. At the end, please say which plan you would choose and why.
17. #users1st
Your mission [20 m]
1. Decide on a website to test (general consumer is best)
2. What do you want to find out?
3. Write tasks for users (desktop only) + one person types them
4. We’ll review a couple (you read them out)
5. Launch them on the platform!
u: workshop1@whatusersdo.com
p: dream
5 x UK users (keep the demographics broad)
18. #users1st
Top 9 Test Types
1. Live site key journey to identify points of friction
2. Pre-live design assets e.g. Axure prototype to de-risk change
3. Cross device RWD
4. How Users Search starting at Google
5. Specific pages/journeys to inform AB Testing
6. Email campaign > online journey testing
7. Competitor benchmarking
8. Scaling Lab insights
9. International local language
We’ll send you 10 Test types with Tasks
19. #users1st
How many participants?
Usually 10 per segment for key journeys
5 if it’s very specific e.g. a specific area of a prototype
Usually multiplied by device types
20+ for Card Sorting/Tree Testing
50+ for user research or marketing type tests
100+ full scale Abandonment Audit, Digital Index
20. #users1st
Choosing participants
Online panel: Demographics + screener
or/and your customers: Private Panel
or/and Special Recruit: anywhere
We’ll send you the Panel Book
23. #users1st
5 x Benefits discussed in the session
1. Users are in their natural browsing environment
2. Results are fast (days not weeks)
3. Videos are compelling when shared internally
4. Tests can be run internationally
5. Less logistical setup than in-person methods
24. #users1st
5 x Objections discussed in the session
1. Testers take many tests and are “semi-professional”
• Counter with: testers are quarantined so they don’t take too many
2. You can’t ask probing questions mid-test
• True, but the higher participant volume mitigates this
3. Unlike eye-tracking we can’t see where users are looking
• You don’t need to & Eye-tracking has a very poor (if any) ROI for UX:
http://www.slideshare.net/harrybr/what-you-need-to-know-about-eye-
tracking-new-uxlx-version
4. It can take a long time to analyse lots of video
• Allow 1 hour per video or ask WhatUsersDo to analyse for you
5. You can’t get the right types of users
• We can - through special recruits or a panel of your customers
25. #users1st
Case Study
3 x Case Studies:
Fashion retailer = to show how UX testing makes sense of analytics
AO.com = shows how UX Testing improves conversions (read online)
Shearings = shows how UX Testing improves marketing (read online)
27. #users1st
AO.com Case Study
Like you they had a development list/product roadmap
Based upon hunches and opinion
Prioritised by HiPPOs
28. #users1st
User eXposure
Matthew Lawson insisted everyone watch users buy white goods
On their site, on competitors’ sites, starting at Google
Listed the top issues and re-wrote the development list
In a few months made changes that:
Increased online sales by 9.5%
Reduced contact by 33%
Increased engagement (product reviews) 12%
Read the full Case Study
33. #users1st
Shearings Case Study
25 users watching both versions
Images changed too rapidly
"Using WUD to A/B test our TV campaigns has been an extremely quick
an efficient way to radically enhance the performance our campaign.”
Chris Barker, Head of Marketing, Shearings
34. #users1st
Getting buy-in
Rehearse the benefits and objections
Use 3 x Case Studies: Analytics, Conversions & Marketing
Use your free WhatUsersDo video
Bonus: call it Digital Customer Experience Testing
36. #users1st
Tag Types: issues
High: conversion killers, that block users or can prevent people from
converting in real life.
Medium: may become conversion killers when they occur more than once.
Low: small niggles that slow down users and decrease trust in the site.
37. #users1st
Tag Types
Positive: highlights in the experience, moments where users go “wow”.
We don’t tag when the interface is just doing what it’s supposed to do.
Suggestions: when users suggest an improvement.
Other: everything else worth noting.
38. #users1st
3 x Tagging Tips
1. Structure Tag names as: Area + Issue e.g.
• Basket – page speed
• Homepage - Images not relevant
• Delivery - Shipping price not clear
• PDP - Detailed information needed
• Compare page - benefit not clear
• Calculator - Difficult interaction
39. #users1st
3 x Tagging Tips
2. Give context to the event described in the tag: for example if the user
struggles with navigation include in the tag the moment when they say
what they are looking for. The video will be easier to share.
3. Allow about 1 hour for each 20 minute video
41. #users1st
6 Steps to Build Presentations
1. Read the WUD platform synopsis [MS clients]
2. Watch playlist + star clips
3. PPT: Replay the test objectives, tasks & participants
4. PPT: Count of issues by severity and type
5. PPT: Exec Summary of 5 main findings
6. PPT: Each issue: problem & recommendation
With a screen shot
With a video clip (from those you starred)
“User quotes”
Written verbosely so non-attendees can understand it
42. #users1st
1. Read the WUD platform synopsis
(Managed Service clients only)
• The written synopsis includes:
• Overall user experience, including trends
• Top 5 positives
• Top 5 issues
• Any user suggestions
44. #users1st
2. Watch playlist + star clips
Watch a playlist of all the video tags from your project, prioritised by
high, medium and low severity issues, positives and user
suggestions.
Favourite the “killer clips” which support the main issues you need
to address.
Download your favourites in a CSV file, or download individual clips,
to embed them into your presentations.
45. #users1st
3. PPT: Replay the test objectives, tasks & participants
Replaying the test objectives and scope
of test is great for reminding those on
the project, as well as informing
stakeholders who might review many
different supplier reports.
It helps “set the scene”.
46. #users1st
4. PPT: Count of issues by severity and type
Screengrab the coloured bars in the
WUD platform, and embed into your
presentation, to illustrate number of
issues and issue weighting (high,
medium and low severity issues, and
positives).
Use icons to visually illustrate what type
of issues caused the most problems for
users.
47. #users1st
5. PPT: Exec Summary of 5 main findings
Briefly summarise the top 4 or 5 issues
from the project which users
experienced, and visually illustrate them
with icons.
48. #users1st
6. PPT: Each issue: problem & recommendation
Expand upon each issue in detail.
Illustrate the issue with:
- Screengrabs
- User quotes
- “Killer” clips
Follow up with recommended Action to
take, and what the expected outcome
would be if that action was taken.
This will give you a really compelling
deck of slides, to overcome stakeholder
objections, and achieve buy-in for
necessary change or further testing.
52. #users1st
Sum-up
We’ve now covered…
getting buy-in for remote testing
know what you can test
design your own tests
launch them
analyse them
and present results internally.
53. #users1st
You can do this…
Self Serve
barend@whatusersdo.com
= DIY, occasional.
Managed Service
nathan@whatusersdo.com
= Expert Team, BAU.
or a blend.
Principles: Natural environment
Users – not UX specialists
Answers ‘ Why’ – Google Analytics ( Qualitative Research - supports Quant)
Process: Users are given a test script which requires them to perform tasks on your website. – We record their spoken thoughts and their action and then use our UX experts to analyse it in a way that provides actionable insight