Your data is great, but does it work for your users

xmlsummerschool.com Slide 1
Licensed under a Creative Commons
Attribution-Noncommercial-Share Alike 3.0 Unported License
summer school
An introduction to user research
Vicky Buser
Information Architect
Your data is great, but does it work for
your users?
16 September 2015

Learning objectives
By the end of this session, you’ll have the answers to
the following 3 questions:
1. Why should I be interested in user research?
2. What tools and techniques are available?
3. How can I extract most value from the research?
XML Summer School 2015 2

Part 1: Why does user research
matter?

What makes a bad website?
Chances are you’ll say it’s one that makes it hard to
find the information you need or difficult to carry out
the transaction you want to make.
Sometimes sites hide useful information under
improbable menu headings, or none at all.

Imagine a supermarket with no labels

Why does this happen?
• The site doesn’t speak the users’ language
• There’s little or no ‘information scent’
• The information is organised in ways that don’t
match the users’ mental models of the domain

What are the costs?
• It can prevent someone from doing their job
properly
• People waste time and money
• They get frustrated
• The site’s credibility is damaged
• The site doesn’t get repeat traffic
• For commercial sites, the company loses money

As Jakob Nielsen says…
“On the web, if a site is
difficult to use, most people
will leave.”

So how do you prevent this?
• Do some user research!
• How many of you have ever been involved in any
user research? What did you do?
• The joy of user research is that there are many
tools and techniques you can use to suit the
specific constraints of your project
• Doing user research is the thing I’ve always
enjoyed most about my job
• Users (people) will always surprise you and act in
unpredictable ways

What we’re going to cover
• Why user research is important for everyone
• What it is
• Tools and techniques
• How to avoid some of the pitfalls
• How to extract most value from your findings

Why is user research important for you?

http://alistapart.com/blog/post/developing-empathy
As a developer recently blogged…

And it’s important to me because…
I’m an Information
Architect.
It’s my job to organise,
label and group content in
ways that help people find
what they’re looking for.
User research is a really
important part of getting
this right.

https://insidegovuk.blog.gov.uk/2015/07/14/rebuilding-browse-based-on-user-needs/

Why user research is crucial in IA
• The way we organise data affects the people who
use it
• There’s not much point in organising things in ways
that don’t help people find what they’re looking for
• People have different ways of looking for the same
information, depending on the context
• Language can be very ambiguous
• How you organise things depends on your
audience

How would you classify these?
• Sweets?
• Confectionery?
• Chocolates?
• Something else?

Language is tricky
• People use different words to describe the same
thing: Courgette/zucchini? Chips/crisps?
Aubergine/eggplant? Coriander/cilantro?
Sweet/desert/pudding?
• Specialists may use different language to the
layperson: Pharmacovigilance/Drug safety?
Residency/Living in the UK?
• The same words can be understood in more than
one way: mercury, fly, fast, fair
• Language is always changing: Chavs, mentally
retarded (examples from BBC Archives)

Context affects how we label things
• IAs try to design labels that speak the same
language as the site’s users
• They’re an important part of the information scent
- the hints a user gets from words/labels
• Labels must be user centric not organisation
centric
• Given that there’s always an element of messy
subjectivity with labelling, it’s important to validate
them with rigorous user research

Back to the developer…

What do I actually mean by user research?
• User research isn’t just usability testing
• It’s about learning the whole picture rather than
just what happens on the day
• It’s an ongoing activity throughout the product
cycle
• It’s investigating and analysing what your users
need to inform your site’s strategy
• And then validating or disproving design
assumptions, such as “Practitioners need different
browse structures to the general public on GOV.UK”

Yes, you say, but I already know about my
users. They’re just like me!
• Although it’s very tempting, you can’t assume your
users are like you
• You are not your user – you’re far too close to the
data/product/content
• It’s all too easy to think that other people think
about things the same way that you do
• But you need to test these assumptions and
provide evidence to support or challenge them

Developing empathy

What do we want to learn?
• About users’ mental models: how do people
currently understand a topic? What kind of picture
have they built for themselves about how a given
task is done or organised?
• What do users consider most important?
• What are the names and relationships between the
terms people use?
• What’s the larger picture of the problem we’re
trying to solve?
• What are the users’ needs?

Overall benefits
• People’s emotional needs and motivations are
more stable over time than shifting technologies,
so having an in-depth understanding of your users
will help you adapt more easily to new technology
• User research is good for getting
consensus/settling arguments: rather than
guessing what your users need, you can find out
• Helps to prioritise features so you don’t waste time
developing features users don’t need

Part 2: Tools and techniques

Framework for the overall process
• Define hypotheses/assumptions to test
• Explore whether your assumptions are correct and
then modify your tests based on your findings
• Act on your findings to improve the site
• Continue to experiment and iterate
• Ask “How can we always get better?”

Mixing and matching the tool kit
• A wide range of methods are available
• Best to mix and match according to need and
combine insights
• Here is a selection of some of the most commonly
used techniques…

Some quantitative methods
• Site analytics
• Search log analysis
• Surveys
• A/B testing

Site analytics
The GOV.UK performance
dashboard shows us the most
popular content, device usage,
seasonal patterns in usage etc.

Site analytics
Great for:
• Identifying interesting patterns or behaviours for
more probing follow-up research
• Measuring improvements (success metrics)
Look for:
• Usage patterns, traffic cycles, key user journeys
through the site, drop-off points, device and
platform usage
Limitations:
• Tells you what users do but not why they do it

Search log analysis
Great for:
• Seeing what your users are looking for expressed in
their own language
• Determining labels that speak the user’s language
• Gathering useful information about what users
want from your site and how they articulate their
needs
Look for:
• Acronyms, jargon, new words, length of search
queries, common misspellings

Search log analysis
For example, looking at terms people
are searching for from specific pages
(on-page searches in Google
Analytics) can highlight problems
when users aren’t finding the specific
information they’re looking for on
that page – e.g. a specific form.

Surveys
Great for:
• Reaching a broad audience
• Finding out about demographics
• Getting lots of data from a large number of users
Limitations:
• Surveys need to be well designed to be effective
(survey design and analysis are specialised fields of
research)
• Best for eliciting facts rather than a deep
understanding of user behaviours

A/B testing
What is it?
Also known as “multivariate testing,” “live testing,”
or “bucket testing”: a method of scientifically testing
different designs on a site by randomly assigning
groups of users to interact with each of the different
designs and measuring the effect of these
assignments on user behavior.

A/B testing
Great for:
• Measuring the live impact of design changes on
key business metrics (# orders, registrations etc.)
• Researching design patterns for small changes to
transactions
Limitations:
• Can be hard to frame as a sensible experiment
• Many ways to do it wrong
• Creates a focus on short-term improvements, so
shouldn’t be used in isolation

A/B testing example
GOV.UK compared a new
design of the Contact
page with the old design.
Of 50,000 people who
came to the contact
page, the new design
helped 1000 more of
them get contact details.
https://gds.blog.gov.uk/2014/05/09/using-ab-testing-to-make-things-better/

Some qualitative techniques
• Contextual inquiry (discovery phase)
• User interviews (discovery phase)
• Personas (discovery phase)
• Usability testing (design and build)

Contextual inquiry
What happens?
Researchers meet with and study participants in
their natural environment.
GOV.UK childcare and parenting example: user
researchers visited 16 families in their homes and
listened to their experiences of becoming a parent.

Contextual inquiry
Great for:
• Observing how users interact with information in
their natural settings – better than asking about it
• Providing environmental context (useful when
physical materials are involved in the process)
Tip:
• Observe people doing their daily tasks and ask
them about their pain points
Limitations:
• Can be time consuming to set up and run

User interviews
What happens?
Researchers meet with participants one-on-one to
discuss in depth what the participant thinks about
the topic in question.

User interviews
Great for:
• Gathering rich, deep understanding about people’s
information needs
• Capturing users’ language and how they refer to
content
Tip:
• Listen to the words people use (jargon, acronyms)
Limitations:
• Can be time-consuming to set up and run
• Sensible to talk to a few people initially and then
consider where to go next

ACTIVITY (20 mins) - interviews
Get into groups of 3 and interview each other about
mental models related to cooking and recipes, based
on this script I’ve prepared. 1 person ask the
questions, 1 person answer the questions and 1
person take notes.
• To the interviewer: remember to listen
• To the note taker: Try to record as many of the
responses as verbatim as possible, so you can re-
use quotes for illustrations
• For the interviewee: Try to be natural and answer
the questions honestly. There are no right or wrong
answers
Take it in turns so that by the end you have three
interview transcripts.

Personas
https://assisteddigital.blog.gov.uk/2014/08/28/assisted-digital-user-personas/

Personas
Great for:
• Simple collaborative exercise to get everyone on the
team to start thinking about users
• Bringing the user to life, developing empathy
• Focusing on target users
• Standing in for real users to guide decisions about
design and functionality
Tip:
• Base your personas on evidence, the details should be
factual – and get the whole team involved
Limitations:
• Not a substitute for talking to real users

Usability testing
(not ‘user testing’!)
Great for:
• Highlighting usability issues and providing
qualitative insights about the problems
• Focus, depth, recordings (if using a lab)
Tips:
• Ask open, neutral questions, get users to ‘think
aloud’, and use self-directed tasks if possible
Limitations:
• It’s an artificial and unnatural environment
• Doesn’t tell you how big a problem is

Only 5 participants? Yes, really…
http://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Pop-up usability testing

GOV.UK childcare example
We did remote un-moderated usability testing with
parents to see how well they could find relevant
information on GOV.UK in it’s previous structure.
We found that parents were visiting 3, 4, sometimes
even 5 categories before finding the right one.

Some qual/quant methods
• Card sorting
• Tree testing
• Diary studies

Card sorting
What happens?
Users are asked to organise
items on cards into groups and
assign category labels to each
group. This helps create or
refine a site’s IA by exposing
users’ mental models.
https://newsignature.com/articles/card-sorting-techniques-pros-cons

Card sorting
Great for:
• Simple, cheap and fun way to learn about how
users group and label information in a specific
domain
• Providing insights into users’ mental models
Tips:
• Look for consistent groups with similar cards in
them and how users have labelled their groups.

We did a remote online card sort of content with 100
parents to see whether the initial groupings we came
up with were consistent with the way parents think
about them.
We looked at how often any two cards were placed
together (similarity matrix) and the 3 card sorts that
had close similarity to the most sorts.
We found that there were no strong groupings
around financial help (consistent with the lack of
knowledge around financial help that came out in
the initial research).

ACTIVITY (15 mins) – card sorting of recipes
Get into teams and sort these recipe cards into
groups that make sense to you.
Then label the groups using the post-its.
Present your findings to the rest of the group at the
end.

Tree testing
What happens?
Users are asked to find information in a simplified
version of the site’s hierarchy. This tests how well
information can be found in the structure.

Tree testing
Great for:
• Measuring how well users find items in a defined
hierarchy
• Testing labels you’re not sure about
Risks:
• Tests how people look for information, not how
they would classify it (that’s what card sorting
does)

We tree-tested the categories we came up with after
the card sorting to check that they made sense and
to work out where content should be duplicated in
more than one category.

Diary studies
What happens?
Participants are given a mechanism (e.g. diary or
camera) to record and describe aspects of their lives
that are relevant to a product or service, or simply
core to the target audience.
Analysing diaries kept by users reveals how mental
models can change with time.

Diary studies
Great for:
• Tracking ongoing relationships with your users over
time
Limitations:
• Diary studies are typically longitudinal and can only
be done for data that is easily recorded by
participants

Part 3: Challenges and extracting
value

Challenges
• Budget
• Planning
• Doing
• Analysis
• Reporting

Budget
“We don’t have any time or money for research!”
• It doesn’t have to be slow or expensive
• Main costs = people’s time, recruiting and paying
participants, equipment costs
• ROI - helps reduce waste on things that don’t work
or aren’t needed
• Remember the costs of users not being able to
find what they need…
• In reality there’s always some form of user
research you can do even on the smallest budget

Planning
Defining the right problems to solve
• Quantitative techniques can help you start
• Be clear about the objectives of your research
Think about when you do your user research
• As early as possible once you have clear goals
• Match the right technique to the right project
stage
• Write a brief for each activity
Think about how the data will be analysed (metrics)
and allow plenty of time for the analysis phase

Doing: recruiting your participants
• You need a representative sample
• Do the recruitment yourself or outsource it
• Recruit as far in advance as possible
• Users need incentives (money, vouchers, cake!)
• Allow for ‘no shows’
How many moderators?
• Good to have 2 people: 1 for note-taking
• If you can take your team to observe the sessions
that’s great

Doing: on the day
Consent and ethics
• You’ll need to get participants’ permission to
record and their consent to use the data you
collect (best to only share this internally)
• Allow the participant to opt-out at any time
• Respect and listen to your participants
Asking the right questions
• Ask open ended questions that encourage
participants to ‘think out loud’
• Instead of asking for preferences or opinions, try
to observe what people do

Analysis
• The process than transforms your research data
into deliverables that can make an impact
• Don’t skimp on this stage
• Go back to the goals you defined in your planning
stage and how you were going to measure success
• Don’t worry too much about statistical significance
• Look for patterns and insights with post-its and
affinity sorting – ideally as a team exercise
• Be realistic about the confidence you can assign to
your results – you can reduce risks but not remove
them completely

Reporting
• User research is only as good as the impact it has,
so it’s important to communicate your findings
• Use quotations and videos as much as possible in
your presentations (remembering to respect users’
anonymity)
• Include subtitles in your video clips if you can
• Include stories about what people actually said
• Show photos if you have any
• You could even use comic strips
• Use wall space for your artifacts (GDS style)

Wrap-up

Summary
• Talk to your users regularly about their needs
• User research doesn’t always happen in a lab
• Identify your target audience
• Define the problem before trying to solve it
• Don’t assume you are the same as your users
• Use research to validate hypotheses
• Design decisions should be based on evidence
• Get all the team involved if you can
• User research is ongoing
• Best to test ideas early and often

Resources
Steve Krug’s books:
- Don’t Make Me Think
- Rocket Surgery Made Easy
Online resources:
https://userresearch.blog.gov.uk/
http://www.slideshare.net/GDSUserResearch
http://www.gv.com/library/user-research

Final thoughts
What difference would user research make to the
products you’re working on?
Remember…
• The costs of users not finding what they need or
struggling to use a site can be high and…
• Talking to users is usually fun and always
interesting!

Here’s something for you to try
Friends and family ‘guerilla’ usability testing
• A fast, easy, convenient technique
• Show a site, a prototype, a sketch…
• To: friends and family, colleagues (who don’t work
on the project), people in cafes etc
• Ask them about it and observe what they do
• Get immediate feedback with minimal overhead
• Develop a better understanding of which parts of
your site need improving
**CAVEAT: However, where possible you’d want to use actual representative of your audience (see
http://www.gv.com/lib/is-it-a-good-idea-to-test-my-product-with-friends-and-family)**

summer school
xmlsummerschool.com Slide 73
Licensed under a Creative Commons
Attribution-Noncommercial-Share Alike 4.0 International License
If you want to contact me here’s my email:
vicky.buser@gmail.com
@vickybuser
Thank you!

Your data is great, but does it work for your users

Recommended

Recommended

More Related Content

Similar to Your data is great, but does it work for your users

Similar to Your data is great, but does it work for your users (20)

Recently uploaded

Recently uploaded (20)

Your data is great, but does it work for your users

Editor's Notes