1.
User-Testing,
Testing, 1, 2,
3…
Webinar Series: User-
Experience

2.
Agenda
• The W’s
• Types of User-Testing
• How User-Testing Fits in the
Web Development Process
– Persona Development
– Information Architecture Testing
(IAT)
– Visual Affordance Testing (VAT)
– User Acceptance Testing (UAT)
– Multivariate Testing (MVT)

3.
the W’s

4.
What?
• What is User-Testing?
User-Testing is a focal part
of the User Centered Design
philosophy that aims to gain
direct feedback from users and
collect actionable data

5.
What?
• What Can We Test?
•Wireframes
•Designs
•Prototypes
•Live Websites
•Everything!

6.
Why?
• Why Test Users?
• We often think we
know what is best for
our users
• You want to add
innovative
functionality to your
site
• You have low
conversion rates
• You have high
customer support

7.
When?
Test Expectations
/ Mental Model
Test User
Benchmark Expectations
Test Visual
Affordance
Test Live
Website
Test User
Performance

8.
Who?
• Who Do We Test?
• Users, not Testers
• Real people who use your
website
• Rinse, Lather, REPEAT!

9.
Who?
• Who Do We Listen
To? We need to place four ad
banners on the homepage
I want to read content
that is fresh and
interesting
We need to drive users
to download our latest
whitepaper
I don’t want my
decisions manipulated
We want to display
company news and our
stock ticker on the
homepage
I know what I want and
want to find it
without being
distracted

10.
Where?
• Where Do We Find Users to Test?
•On your website
•Through a recruiter
•In their natural
environment

11.
How?
• How can we avoid having to
figure all this out on our own?
Let the users do
all the hard work
for you!

12.
Types of User-Testing

13.
Types of User-Testing
• Remote
• In Lab
• Automated

14.
Types of User-Testing:
Remote
• Users sit in
their natural
environment
• Users are asked
to carry out
specific tasks
using prototypes
• Their mouse
movements and the
phone
conversation are
recorded for

15.
Types of User-Testing: In
Lab
• Video camera
records users’
facial expressions
• A screen recorder
records their
screen movements
• A facilitator is in
the room with them
• Observers are in
another room,
usually looking
through a 2 way

16.
Types of User-Testing:
Automated
• Users are
intercepted when
they appear on a
site
• Users are asked to
fill out a short
survey before
navigating the site
• All the user’s
movements are
recorded while
navigating the site
• Users complete an
exit survey when
they leave the site

17.
Types of User-Testing
Type Pros Cons What to Use it On
Remote Nationally diverse geographic reach Lose the ability to see users’ E-commerce web sites
expressions and gestures
Quick Large, informational web sites
Unnatural environment
Less Costly Web Applications
Intranets
In Lab Obtain data based on user’s body More costly Highly secure client/server
movement and facial expressions applications
Lack of geographic reach
Handheld Apps or other
products with a significant
hardware component
Automated Captures users that come to the site Steps in between start and goal E-commerce web sites
and records users’ activities is not elicited
Large, informational web sites
Natural environment
Web Applications
Actual users
Intranets

18.
Types of Data Collected
• Quantitative • Qualitative
– Statistics, – Opinions,
Trends Details
– Large Sample Size – Small Sample
– Structured Data Size
Collection – Unstructured
Methods Data Collection
Example:
Methods
Example:
•30% of users clicked on the red ―I tried clicking on the button
button because the white text on the
•70% of users clicked on the blue blue background caught my eye
button and the text was large‖

19.
Informal vs. Formal
Testing
• Informal • Formal
– Casual – Structured
– 3-4 Days – 3-4 Weeks
– In Your Office – In a Lab
– Friends & Family – Broad Range of
– Paper Prototypes Users
– Less Common – High Fidelity
– More Common
– Bias
– Incorrect – Trained Moderators
Analysis & Analysts

20.
POLL
• Have you conducted any type of
User-Testing on your current
website?

21.
How User-Testing Fits in
the Web Development
Process

22.
The Web Development Process
Test Expectations
/ Mental Model
Test User
Benchmark Expectations
Test Visual
Affordance
Test Live
Website
Test User
Performance

23.
The Testing Methods
Card Sorting
Wireframe
Persona Testing
Development
Visual
Affordance
Testing
Multivariate
Testing
User
Acceptance
Testing

24.
Persona Development (PD)

25.
PD:
Card Sorting
Wireframe
Persona Testing
Development
Visual
Affordance
Testing
Multivariate
Testing &
Optimization
User
Acceptance
Testing

26.
PD:
Q: How do we develop a website that
is tailored to our users needs,
behaviors, experience, and
technology?
A Ask
them!
:

27.
PD: What is a Persona?
• A persona tells the story about
the users of your website and
attributes lifelike features to
make them more relatable
• Based on real data, the persona
serves as a tool to help everyone
involved in development
understand who and what the
website is for
• The persona creates a shared
vision

28.
PD: Market Segments vs.
Personas
Market Segment
• Gender: 75% Female, 25% Male
• Region: 99% in United States
• Size: 70% metro, 20% suburban, 10% rural
• Generation: 10% baby-boomers, 20% Gen-X, 70% Gen-Y
• Income: 25k – 50k
• Occupation: Student, Administrative Assistant,
Intern
• Education: High School Diploma
• Family Lifecycle: Single

29.
PD: Market Segments vs.
Personas
• Persona

30.
PD: Surveys & Interviews
Surveys Interviews (contextual +
• Quantitative Method individual)
• Need about 400 • Qualitative Method
responses to account • Need about 8 participants
for margin of error per segment

31.
PD: Benefits
• Focus on the users’ goals and
needs throughout the development
process
• Prioritize features, design,
efforts based on what is most
needed by the personas
• Any internal battles will can be
settled by referring to the
personas

32.
PD: Tips for Success
• Always include:
– A Name
– A Photo
– Demographics
– Description based on data collected
– Goals
– Common tasks carried out on the website
– Habits
• Always refer back to the personas!

33.
Information Architecture
Testing (IAT)

34.
IAT: Card Sorting
Card Sorting
Wireframe
Persona Testing
Development
Visual
Affordance
Testing
Multivariate
Testing &
Optimization
User
Acceptance
Testing

35.
IAT: Card Sorting
Q: How do we ensure that the
taxonomy and site linking
structure are intuitive for our
users to find our products and
services?
A Have the user label
and organize pages
: for you!

36.
IAT: What is Card Sorting?
• Users are given ―cards‖ with
labels on them and asked to
categorize the information on
your site into meaningful
groupings.
• Method: In-Lab or Automated
• Sample Size: At least 15 per
segment

37.
IAT: Types of Card Sorting
Open Card Sorting Closed Card Sorting
• Categories and • Fixed categories are
category labels can provided to the user
be created or and they may not re-
changed by users label or created new
categories.
• Used early during
the development
cycle • Used later in
development cycle
• More difficult to for further testing,
analyze data or to add new
content to the
website
• Easier to analyze
data statistically

38.
IAT: Card Sorting Tools
(Lo-Fi)

39.
IAT: Card Sorting Tools
(Online)
• Online tools allow for easier
data collection / analysis and
larger sample sizes

40.
POLL
• What is your primary method for
conducting card sorting?

41.
IAT: Card Sorting Benefits
• Easy to administer
• Sets the foundation for the website’s navigation
and structure
• Intuitive taxonomy in the users’ ―natural
language,‖ and matches what users are searching for
(SEO)
• Intuitive taxonomy in the users’ ―natural
language,‖ and matches what users are searching for
(SEO)

42.
IAT: Card Sorting Example

43.
IAT: Card Sorting Tips for
Success
• If in-lab, ask users to talk out
loud
• Limit the study to a reasonable
amount of cards (~50)
• Note the relationships between
cards and situations that indicate
cross-linking between categories
will be helpful for users

44.
IAT: Wireframe Testing
Card Sorting
Wireframe
Persona Testing
Development
Visual
Affordance
Testing
Multivariate
Testing &
Optimization
User
Acceptance
Testing

45.
IAT: Wireframe Testing
Q: How do we find out if the layout
of information on the site is
intuitive, and if the placement of
images, text, links, messaging, and
calls-to-action help users find
what they need?
A Watch users as they
navigate through
: the wireframes!

46.
IAT: What is Wireframe
Testing?
• Wireframe testing is a usability test of
the prototype pages of your site
excluding colors and other details. This
involves creating realistic tasks and
having users navigate the prototype to
fulfill those tasks.

47.
IAT: Wireframe Testing
• Method: In-Lab or Remote
• Sample Size: At least 5 per
segment
– Why 5 Users?

48.
IAT: Wireframe Testing
Example
• You are out in the
field and you need to
find information on
how to troubleshoot
the network. Where
would you start?
• Your business has just
switched over to VoIP.
Where would you go to
look for information
on how to manage your
new VoIP network?

49.
IAT: Wireframe Testing:
Benefits
• Find out if the layout of the page
achieves your users’ goals and your
goals
• Low cost and quick to make changes
to a wireframe versus a real
website

50.
IAT: Wireframe Testing: Tips
for Success
• Create tasks that are real
situations for the user (refer to
the personas)
• Do not bias the participant by
leading or priming
• Be patient!

51.
Visual Affordance Testing
(VAT)

52.
VAT:
Card Sorting
Wireframe
Persona Testing
Development
Visual
Affordance
Testing
Multivariate
Testing &
Optimization
User
Acceptance
Testing

53.
VAT:
Q: How do we make sure the design is
not only aesthetically pleasing,
but also that messaging and calls-
to-action are clear for the user?
A Ask users, and
watch as they
: navigate through
the designs!

54.
VAT: What is Affordance?
• Affordance is a quality of an
object, or an environment, that
allows an individual to perform
an action.
• A visual clue to the function
of an object

55.
VAT: What is Visual
Affordance Testing?
• Once images, colors, and other
design elements have been
implemented based on the
wireframes, VAT requires users to
indicate what they perceive as
clickable on the design
• Method: In-Lab or Remote
• Sample Size: At least 5 per segment

56.
VAT: Benefits
• Validates if key calls-to-action
are even seen by users
• Reveals elements of a design which
may mislead users to believe are
clickable
• Reduce user frustration by
addressing problems found in the
VAT

57.
VAT: Example

58.
VAT: Tips for Success
• Pay close attention to any
hesitation or vocal expressions
• Visual affordance does not only
apply to calls-to-action or links

59.
User Acceptance Testing
(UAT)

60.
UAT:
Card Sorting
Wireframe
Persona Testing
Development
Visual
Affordance
Testing
Multivariate
Testing &
Optimization
User
Acceptance
Testing

61.
UAT:
Q: How do we find out if all the
content, interactions, and media
on the prototype we’ve built will
resonate with our visitors, and if
they negatively affected our
site’s usability?
A Ask users, and watch as
they navigate through the
: prototype!

62.
UAT: What is User Acceptance
Testing?
• User Acceptance Testing is a
usability test conducted on a
fully functional prototype
prior to launching the website
• Method: In-Lab or Remote
• Sample Size: At least 5 per
segment

63.
UAT: Benefits
• Find out how users will really
use the website
• Figure out if all functionality
and interactivity is useful to
the user (Flash, Videos, AJAX)
• Observe where people may have
difficulty or fail and how they
recover from those errors

64.
UAT: Tips for Success
• User Acceptance Testing is not
Quality Assurance!
• Allow for free exploration with
the user thinking aloud
• Test iteratively

65.
POLL
• Have you added any new content
to your website in the past 6
months (whitepapers, downloads,
videos)?

66.
Multivariate Testing (MVT)

67.
MVT:
Card Sorting
Wireframe
Persona Testing
Development
Visual
Affordance
Testing
Multivariate
Testing &
Optimization
User
Acceptance
Testing

68.
MVT:
Q: How do we make sure that our
site is always usable and that the
user-experience of our visitors is
always optimal?
A Continue to test and
optimize your site!
:

69.
MVT: What is Multivariate
Testing?
• Once the website is launched,
Multivariate Testing optimizes
performance of the website by
segmenting visitors and displaying
different images/ buttons/ links/
content to them
• Method: Automated, In-Lab, Remote
• Sample Size: At least 500 visitors

70.
MVT: A/B Testing vs.
Multivariate Testing
A/B MVT
Landing Page Version 1 Landing Page Version 2 Landing Page Element Views Form Submissions Conversion Rate
Views 185,854 187,900 1C - Banner [static image] 1,300 517 39.77%
Form Submissions 4,154 5,232 2A - Button ["Buy Now"] 1,287 270 20.98%
Conversion Rate 2.24% 2.78% 3B - Text [black] 1,340 245 18.28%
2C - Button ["Submit"] 1,310 230 17.56%
1B - Banner [AJAX] 1,305 125 9.58%
3A - Text [gray] 1,290 105 8.14%
3C - Text [blue] 1,297 105 8.09%
2B - Button ["Next"] 1,304 99 7.60%
1A - Banner [Flash] 1,289 88 6.83%

71.
MVT: Benefits
• Minimize risk with any website
changes
• Data drives decisions
• Data is collected 24/7
• Tangible Results
• Test New Applications
• Test the ―Third Screen‖

72.
MVT: Tips for Success
• ABT – Always Be Testing!
• Preparation and set-up is
crucial

73.
What Next?
• Rinse, Lather, REPEAT!

74.
―If ease of use was the only valid
criterion, people would stick to
tricycles and never try bicycles‖
Doug Engelbart
Human-Computer Interaction Pioneer
and inventor of the computer mouse

75.
Thank You!
Jegan Chen, Usability Engineer
BusinessOnLine
Jegan.Chen@BusinessOL.com
http://www.businessol.com
Additional BusinessOnLine
Resources:
Re:Cognition: BusinessOnLine’s Usability Blog
http://www.businessol.com/usability_blog
Upcoming Webinar:
SEO Webinar Series:
Session 3: Not Just Video—VideoSEO
June 17, 2009 11am PST (2pm EST)
http://www.businessol.com/news/main-webinars/