A practical approach to evaluating website performance, focusing on: 1. How to isolate usability issues and identify conversion rate improvements 2. Website testing methodologies (A/B & MVT) and how to start an in-house testing program 3. Some Conversion Rate Optimisation tips

1. Taking
The
Guesswork
Out
Of
Website
Op6misa6on
(CRO)
[A
business
case
for
Digital
Improvement]
January
2014
Nathan
Raward

2. Focus
of
this
session…
1.
Outline
a
structured
approach
to:
• Evalua6ng
website
performance
• Isola6ng
usability
issues
• Iden6fying
conversion
rate
improvements
2.
Provide
an
overview
of:
• Website
tes6ng
methodologies
(A/B
&
MVT)
• How
to
start
an
in-­‐house
tes6ng
program
3.
Share
some
Conversion
Rate
Op6misa6on
6ps:
• Key
op6misa6on
themes
• Specific
elements
for
tes6ng

3. Taking
The
Guesswork
Out
Of
Website
Op6misa6on…
1.
SEO
&
adver6sing
will
bring
users
to
your
site
but
will
they
convert…?
2.
Fixing
underlying
usability
issues
will
improve
the
conversion
baseline
4.
An
established
CRO
program
can
help
mi6gate
guesswork
altogether
3.
Running
ini6al
A/B
tests
will
help
to
reveal
conversion
levers
Website
Op6misa6on
relies
on
validated
learning
(data,
insights
&
experience)
to
de-­‐risk
design
decisions
and
provide
a
higher
probability
of
success

4. Validated
learning
helps
us
to
avoid
these
familiar
scenarios…
“Hi.P.P.O
Effect”:
Doing
what
the
boss
wants
–
not
what’s
best
for
customers
“Form
over
Func:on”:
When
aesthe6cs
become
more
important
than
conversions
“Paralysis
by
Analysis”:
Relying
on
data
sources
that
are
too
subjec6ve
or
not
representa6ve
of
actual
customers
www.dilbert.com

5. Why
Op6mise?:
To
be
compe66ve
>
mediocre
is
not
good
enough…
Industry
• Online
ad
spend
>$1
billion
for
the
first
6me
in
Q3
2013.
A
25%
increase
vs
2012
(6)
• Mobile
adver6sing
represented
11%
of
total
• 200%
increase
in
new
online
store
openings
vs
2010
(2)
Technology
• Smart
phones
&
tablets
now
represent
38%
of
daily
UVs
(7)
• Shin
from
apps
to
responsive
/
adap6ve
design
and
Paid
apps
on
the
decline
(9)
• Big
Data,
Cloud
Compu6ng
etc.
(power,
scale
&
flexibility)
(1)
(2)
(3)
(4)
(5)
(6)
Users
• 94%
Australians
have
internet
access
(6)
&
>70%
have
bought
online
(3)
• $15bn
spent
online
in
2013
(1)
• Average
Australian
carries
2.6
devices
(8),
while
over
3.3M
are
mobile
only
users
(4)
Website
Owners
/
Company
• 74%
believe
UX
is
key
for
improving
sales
(5)
• 94%
say
personalisa6on
is
cri6cal
(5)
but
only
25%
integrate
their
DB
&
site
(3)
• 70%
of
those
conduc6ng
A/B
tests
saw
CVR%
increase
(5)
hpp://www.theaustralian.com.au/business/australian-­‐shoppers-­‐spent-­‐15-­‐billion-­‐online-­‐report/story
hpp://www.cmo.com.au/mediareleases/16749/new-­‐data-­‐explains-­‐booming-­‐australian-­‐e-­‐commerce/
hpp://econsultancy.com/blog/62132-­‐australian-­‐ecommerce-­‐nine-­‐stats-­‐you-­‐might-­‐have-­‐missed
hpp://www.digitalbusiness.gov.au/2013/07/16/australians-­‐cut-­‐the-­‐cord-­‐the-­‐rise-­‐of-­‐mobile-­‐only-­‐users/
hpp://econsultancy.com/blog/64095-­‐100-­‐amazing-­‐stats-­‐from-­‐econsultancy-­‐s-­‐2013-­‐reports
hpp://iabaustralia.com.au/-­‐/media/IAB/Files/OAER%20exe%20summary/PwC%20IAB%20OAER%20Sept
%202013%20exe%20summary.ashx
(7)
(8)
(9)
hpp://www.iabaustralia.com.au/en/Insights/Resource_Library/IAB_BP_and_Guidelines/-­‐/media/
2444182B69B34BDF9A91C01EBE71A8A9.pdf
hpp://nakedsecurity.sophos.com/2013/03/14/devices-­‐wozniak-­‐infographic/
hpp://techcrunch.com/2013/07/18/paid-­‐apps-­‐on-­‐the-­‐decline-­‐90-­‐of-­‐ios-­‐apps-­‐are-­‐free-­‐up-­‐from-­‐80-­‐84-­‐
during-­‐2010-­‐2012-­‐says-­‐flurry

6. So
what
does
a
good
website
look
like?
What
Customers
want
What
Companies
want
Website
that
is
easy
and
fun
to
use
Best
product,
price,
delivery,
service
More
traffic
&
customers
More
sales
&
conversions
Relevant,
engaging
&
targeted
content
All
day,
everyday
access
to
support
More
sales
/
customer
(LTV)
Lower
CPA
&
cost-­‐to-­‐serve
Joined-­‐up
experiences
Personalised
experiences
Best
Omni-­‐channel
experiences
Defensible
differen6a6on
Finding
the
right
balance
between
what
customers
and
companies
want
plus
the
mastery
of
several
founda6onal
/
transforma6onal
approaches:
Analy6cs
&
Insight
User-­‐Centred
Design
Sonware
Development
Flexible
&
Scalable
Platorms
Op6misa6on
‘Test
&
Learn’

7. Ok
–
so
what
does
a
website
look
like?

8. Website
Op:misa:on
Process:
How
to
get
to
good…
Clarify
Website
Purpose
&
Proposi6on
Define
Business
Outcomes
&
Baseline
Website
Analysis
(Quan6ta6ve,
Qualita6ve
&
Capabili6es)
Isolate
issues
&
iden6fy
improvements
Rank
&
Priori6se
Website
Changes
Design,
Iterate
&
Validate
Test!
Test!
Test!
(Op6mise)
Learn,
Report,
Repeat

9. Clarify
Website
Purpose
&
Proposi:on
Objec:ves:
• What
is
the
purpose
of
the
site?
• What
does
it
do?
• Who
is
it
for?
• What
customer
need/s
does
it
serve?
• What
is
the
core
value
proposi6on?
Outcomes:
• Valida6on
of
whether
the
intent
of
the
website
&
value
proposi6ons
are
communicated
clearly
and
in
an
appropriate
way
for
the
target
audiences
(&
a
list
of
ini6al
improvement
ideas…)

10. Define
Business
Outcomes
&
Baseline
•
•
•
•
Define
Success
metrics
(KPI’s)
that
directly
or
indirectly
impact
business
performance
Examples;
bounce
rate,
visits,
transac6ons,
revenue,
AOV,
CVR%
(mul6ple)
Develop
a
scorecard
to
show
performance
vs.
target
&
prior
week,
month
&
year
USE
IT!
ILLUSTRATIVE

11. Website
Analysis
(Quan:ta:ve)
• Visualize
user
flows
across
key
journeys
&
conversion
funnels
(Web
Analy6cs)
• Iden6fy
problem
areas;
where
users
bounce,
drop-­‐out
or
don’t
convert
1
Bounce:
xx%
Journey
Objec:ve
Grab
apen6on
Browse
Conversion
Objec:ves:
Improve
site
s6ckiness
Step
Home
page
xxx
(?%)
Drop-­‐off:
2
?%
Review
/
Filter
Category
Pages
Increase
intent
Product
Pages
xxx
(?%)
?%
Incen6vise
Add
to
basket
xxx
(?%)
?%
xxx
(?%)
3
Facilitate
Checkout
xxx
(?%)
?%
Reassure
Buy
Convert
more
browsers
into
buyers
Purchase
xxx
(?%)
?%
[Simplified
e-­‐commerce
view:
many
journeys
will
not
be
so
linear]
Reduce
cart
/
checkout
abandonment

12. Website
Analysis
(Qualita:ve)
UX
Audit:
Template-­‐based
checklists
can
help
Expert
(Heuris:cs)
Review:
iden6fy
usability
to
iden6fy
major
usability
issues
issues
and
opportuni6es
for
improvement
1
2
3
TECHNICAL:
DESIGN:
NAVIGATION:
4
5
6
CONTENT:
E-­‐COMMERCE:
AVOIDANCES:
simple
intui6ve
engaging
suppor6ve
accessible
efficient
Econsultancy
Usage
Scenarios
&
Task
Analysis:
Define
scenarios
&
step-­‐thru
tasks
to
isolate
issues
Benchmarking:
Review
compe6tor
websites
to
iden6fy
gaps
and
generate
ideas

13. Website
Analysis
(Capabili:es
Assessment)
To
be
successful
it’s
essen6al
to
have
the
right
mix
of:
People
Processes
Platorms
A
website
is
only
as
good
as
its
developers,
UX
team
etc:
Without
effec:ve
process
there
is
no
progress:
Technology
underpins
great
experiences:
• Make
sure
you
have
the
right
skills
to
deliver
• Embed
User-­‐Centred
Design
(UCD)
prac6ces
within
agile
/
waterfall
• Device
agnos6c
(flexible)
• Ability
to
scale
&
integrate
• Performance
(load
speed)
• Data
processing
• Do
a
diagnos6c
scan
• HTML5,
CSS3,
Jscript
etc.
• UX,
Designers,
strategists
• Establish
a
‘Test
&
Learn’
culture
(fail
fast)

14. Isolate
Issues
&
Iden:fy
Improvements
Issues:
for
immediate
remedia6on
where
the
solu6on
&
benefit
is
clear
(the
‘no-­‐brainers’)
Improvements:
where
the
benefit
needs
to
be
validated
(tested!)
before
implementa6on
Usability
(fix)
•
•
•
Making
websites
more
user-­‐friendly
&
intui6ve
to
help
customers
more
easily
complete
key
tasks
and
navigate
the
site
Improvements
are
logical
and
can
be
validated
using
qualita6ve
techniques
in
test
/
pre-­‐produc6on
environments
Conversion
gains
generally
moderate
Op6misa6on
(test)
•
•
•
Understanding
how
content
influences
user
behaviour
in
order
to
maximise
pre-­‐
determined
business
outcomes
Improvements
are
based
on
hypotheses
that
require
valida6on
using
quan6ta6ve
techniques
in
a
live
environment
Conversions
gains
can
be
significant

15. Apemp6ng
to
op6mize
conversion
rates
before
fixing
major
usability
issues
is
as
pointless
as…
Fix
your
website
dude!

16. No
really
-­‐
apemp6ng
to
op6mize
conversion
rates
before
fixing
major
usability
issues
is
as
pointless
as…
Fix
your
website
dude!

17. Rank
&
Priori:se
Website
Changes
HIGH
MEDIUM
LOW
User
Experience
impact
VERY
HIGH
Assessing
the
rela6ve
importance
of
changes
based
on
their
an6cipated
impact
on
user
experience
/
conversions
and
the
effort
required
to
implement
them:
4
n
8
n
n
10
n
n
n
Low
Priority
VERY
DIFFICULT
n
DIFFICULT
4.
MEDIUM
Implementa1on
complexity
9
Prominent
value
prop
5.
Live
Chat
Social
Login
New
template
Accordion
menus
9.
n
n
n
6
Add
reviews
7.
7
Reposi6on
CTA
3.
5
Single
page
checkout
2.
2
1.
8.
3
n
n
Top
10
Changes
6.
n
n
High
Priority
1
CTA
message
10. Improved
copy
n
EASY
[ILLUSTRATIVE]
n
FIX
n
TEST

18. Design,
Iterate
&
Validate
The
extent
to
which
any
changes
will
impact
a
websites
visual
design
will
determine
what
artefacts,
itera6on
&
level
of
user
valida6on
is
required:
Paper
sketch
Usability
Low
fidelity
High
fidelity
Working
prototype
high
usage
Implement
low
Op6misa6on
high
usage
low
Test!

19. Test!
Test!
Test!
|
Think
BIG
Start
Small
(A/B)
A/B
Tes6ng
Mul6variate
Tes6ng
(MVT)
vs
BUY
NOW
A
(50%
traffic)
B
(50%
traffic)
BUY
NOW
2^3
=
8
varia6ons
(12.5%
traffic
each)
•
Can
be
run
on
websites
with
moderate
traffic
•
High
traffic
requirements
(depending
#
vars)
•
Test
2
varia6ons
at
a
6me
(control
vs
challenger)
•
Test
poten6ally
thousands
of
varia6ons
(treatments)
•
Good
for
tes6ng
broad
concepts
/
themes
•
Good
for
tes6ng
granular
elements
•
Planning
&
resources
required
=
LOW
•
Planning
&
resources
required
=
HIGH(ish)
•
Provides
limited
insights
into
why
winning
version
is
beper
(effects
confounded)
•
Allows
individual
contribu6on
of
website
elements
on
CVR%
to
be
measured
ALWAYS
test
vs.
the
control
(original)!

20. Test!
Test!
Test!
|
A
Quick
Overview
of
MVT
A
brief
history
of
MVT:
• 1930’s:
Design
of
Experiments
(DoE)
stats
method
developed
by
Sir
Ronald
A
Fisher
Types
of
MVT:
Full
factorial
• 1950’s:
Manufacturing
process
applica6on
developed
by
Genichi
Taguchi
• Show
all
possible
test
varia6ons
• 1990’s:
First
applied
in
an
online
marke6ng
• Detect
main
&
context
(website
experiments)
interac6on
effects
Frac6onal
factorial
• Show
a
subset
of
test
varia6ons
• Confounding
issue
• Taguchi
method
Vendor
Profiles:
From
the
big
4
to
00’s
players
Vendor
selec:on
criteria:
Bought
Omniture
in
09
who
bought
Offerma6ca
in
07
Service
level
Full,
self
or
par6al
(local
support?)
Cost
Enterprise,
PAYG
Bought
Memetrics
&
Maxamine
in
2008
Deployment
Client-­‐side,
server-­‐side
Bought
Interwoven
in
09
who
bought
Op6most
in
07
Methodology
Taguchi,
Choice
modeling,
proprietary
Interface
(ease)
HTML,
WYSIWYG
Bought
Widemile
in
2009
hpp://www.conversion-­‐rate-­‐experts.com/split-­‐tes6ng-­‐sonware/
Integra6on
Web
Analy6cs,
3rd
party
s/ware,
CRM
Segmenta6on
Browser,
cookie
parameters,
custom

21. Test!
Test!
Test!
|
How
To
Run
Tests
In-­‐house
The
Team
Op:misa:on
lead
• Runs
experiment
w/shops
• Manages
tes6ng
roadmap
The
Process
Plan
• Test
page
• CRO
outcomes
Stats
Lead
/
Analyst
• Iden6fies
areas
for
tes6ng
• Analyses
experiment
results
• Hypotheses
• Experiment
design
• Workshop
Web
/
UX
/
UI
Designers
• Creates
mockups
/
concepts
• Develops
test
varia6ons
Tech
/
implementa:on
lead
• Server-­‐side
coding
• Deploys
experiments
• [advanced
experiments
only]
Setup
• Develop
crea6ve
elements
• Verify
treatments
• Tracking
Run
• Run
un6l
sign.
result
achieved
• Minimum
of
2
weeks
(ex.
A/B)
• Avoid
seasonality
Analyse
• Analyse
results
• Sense-­‐check
findings
• Present
results
• Archive
results
Experiment
design
/
dura:on
is
determined
by:
• Available
sample
(traffic
&
base
CVR%)
• Sta6s6cal
confidence
(probability
of
detec6ng
an
effect
in
the
sample)
typically
90-­‐95%
• Rela6ve
effects
(%
difference
between
vars
&
control)
• Number
of
varia6ons
tested
(depth
&
breadth)
Sample
size
calculator:
hpp://visualwebsiteop6mizer.com/ab-­‐split-­‐test-­‐dura6on/

22. CRO
Tips
|
Key
Op:misa:on
Themes
Although
each
company’s
website,
offerings
and
users
are
different,
the
following
themes
/
hypotheses
can
be
applied
to
improve
most
online
experiences
&
conversion
rates:
Optimisation goal
Benefits to users
Benefit to company
Journeys
Help
users
get
from
A
to
B
as
quickly
and
effortlessly
as
possible
More
streamlined
experiences
• Lower
bounce
rates
• Lower
abandonment
Templates
Present
content
in
a
way
that
allows
it
to
be
easily
seen,
read
and
understood
Improved
presenta6on
of
content
• Higher
engagement
• Longer
sessions
Calls
to
Ac6on
Provide
clear
signposts
that
help
users
to
navigate
through
the
site
and
complete
desired
outcomes
Clearer
indica6ons
of
where
&
what
to
click
• Lower
abandonment
• Increased
CVR%
Content
Provide
relevant
and
engaging
content
to
increase
purchase
intent
&
help
virtualise
product
ownership
More
relevant,
engaging
&
targeted
experiences
• Longer
sessions
• More
repeat
visits
Features
Provide
features
that
help
users
to
understand
the
available
op6ons
and
make
informed
decisions
Beper
comprehension
of
content
• Higher
engagement
• Increased
CVR%
Offers
Incen6vise
users
with
offers
that
make
the
decision
to
complete
the
desired
outcome
more
appealing
Greater
perceived
value
&/or
benefits
• Increased
revenue
• X-­‐sell,
up-­‐sell,
AOV

23. Templates
Journeys
CRO
Tips
|
Ideas
For
Tes:ng
(ALWAYS
vs.
control)
Streamline
user
flows
(Create
simple
/
linear
/
logical
journeys)
• Journey
length:
try
removing
any
steps
that
appear
redundant
• Links:
avoid
new
pages,
tabs,
windows
for
more
info
(use
accordions,
tool6ps
etc.)
• SEO
Landing
Pages:
must
feed
into
conversions
journeys
(to
minimise
bounces)
• Naviga:on:
Breadcrumbs
/
Back
&
Next
/
Show
‘n’
/
Lazy
Loading
Minimise
path
to
purchase
(Single-­‐page
checkout
&
floa6ng
cart)
• Form
length:
test
a
shorter
form
(above-­‐fold)
• No.
of
steps:
try
3,4,5
steps
with
all
but
1st
step
collapsed
/
inac6ve
• No.
of
fields:
try
removing
redundant
fields
e.g.
addi6onal
phone
numbers
• Guest
checkout:
allow
users
to
skip
account
crea6on
Design
for
the
fold
(768
pixels)
• Hero
image:
test
a
shorter
banner
to
surface
more
content
• CTA:
posi6on
main
CTAs
above
the
fold
• Accordions:
try
bringing
more
sec6ons
above
the
fold
• Scroll:
intersect
the
fold
with
content
so
users
know
they
have
to
scroll
Do
more
with
less
(Reduce
cluper)
• Content
volume:
test
fewer
content
sec6ons
(7
+/-­‐
2)
to
reduce
cogni6ve
load
• Content
density:
try
using
images,
bullets,
accordions
instead
of
heavy
text
• Visual
hierarchy:
try
using
borders,
shading
&
white
space
to
priori6se
content
• Product
pages:
vary
posi6on
&
size
of
content
allocated
to
copy,
images
&
videos

24. Content
Call
to
Ac:on
CRO
Tips
|
Ideas
For
Tes:ng
(ALWAYS
vs.
control)
Increase
prominence
(&
visibility
rela6ve
to
other
content)
• Colour:
try
more
prominent
colours
that
break
the
standard
palepe
/
theme
• Size:
try
larger
bupons
to
capture
more
real
estate
/
apen6on
• Format:
try
shapes,
shadow,
gradients
etc.
that
dis6nguish
CTA’s
• Posi:on:
must
be
above-­‐fold,
also
test
len
vs.
right
in
containers
• Message:
test
tailored
message
for
ac6on
required
vs.
generic.
Also
synonyms
• Quan:ty:
if
mul6ple
CTA’s
test
giving
primary
CTA
more
prominence
Value
Prop
(sell
the
sizzle!)
• Keywords:
try
inser6ng
key
benefit
&
USP
statements
on
Home
Page
&
key
steps
• Tagline:
test
a
concise
statement
near
logo
e.g.
‘Australia’s
cheapest,
largest
etc.’
• Wrapper:
try
USP
icons
within
site-­‐wide
header
that
link
to
‘About
us’
(reassure)
• Hero
banner:
test
using
hero
(first
rota6on)
to
highlight
USP’s
Copy,
images
&
Video
(quality
not
quan6ty)
• Copy
length:
test
varying
lengths
of
copy
and
use
bullets
to
break
up
text
blocks
• Language:
try
different
styles
e.g.
informa6ve,
humorous,
emo6ve
(human)
• People
images:
test
using
staff,
customers,
people
using
product
(vs.
just
products)
• Product
images:
try
more
images,
360
rota6on,
zoom
(before
applying
site-­‐wide)
• Video:
test
different
themes
e.g.
factual
vs.
humour
also
professional
vs.
staff

25. Features
Simplify
the
decision
making
process
(increase
user
intent)
• Reviews,
endorsements,
awards
&
tes:monials:
test
the
rela6ve
impact
of
each
• Comparison
tools:
try
varying
the
number
&
order
of
apributes,
icons
vs
text
etc.
• Calculators
/
simula:ons:
test
displaying
typical
/
pre-­‐populated
scenarios
• Support
tools:
test
the
following
on
vs.
off;
Click-­‐to-­‐chat,
click-­‐to-­‐call,
Phone
No.
• Intro
/
welcome
videos:
test
instead
of
hero
banners
to
provide
company
overview
• Security
badges:
test
including
badges
e.g.
McAfee,
Verisign
in
cart
&
checkout
• Naviga:on:
test
including
icons
/
images
with
text
to
speed
up
cogni6on
• Sort
order:
test
changing
the
default
sort
order
of
products
e.g.
lowest-­‐highest
Offers
CRO
Tips
|
Ideas
For
Tes:ng
(ALWAYS
vs.
control)
Increase
perceived
user
benefits
• Savings:
show
‘%
off’
and
saving
in
$
in
addi6on
to
price,
test
using
different
fonts
• Urgency:
add
urgency
messaging
‘Hurry
XXX
ending
soon’
but
don’t
overuse
it
• Benefits:
Show
product
/
service
benefits
in
a
table
to
encourage
up-­‐sell
• Compe::on:
show
incremental
value
vs.
compe6tors
in
a
table
or
messaging
Incen6ves
to
assist
conversions
• Free
trial:
test
free
access
for
a
limited
period
(measure
%
that
migrate
to
paid)
• Free
download:
similar
to
above
but
test
a
par6al
download
/
limited
features
• Vouchers:
test
single-­‐use
or
6me-­‐limited
vouchers
to
encourage
repeat
purchases
• Referrals:
test
referral
discounts
to
encourage
users
to
recruit
new
customers

26. Learn
>
Report
>
Repeat
ALWAYS:
Measure
the
impact
of
any
changes
against
the
baseline
/
scorecard
etc.
ALWAYS:
Check
that
the
findings
can
be
explained
logically
(if
not
dig
deeper)
Learn
• Consolidate
findings
• Document
what
worked
&
what
didn’t
• Archive
in
knowledge
base
Report
• Present
results
to
key
stakeholders;
hypotheses
tested,
outcomes
&
results
• Provide
overview
of
next
test
Repeat
• Re-­‐priori6se
roadmap
if
based
on
results
further
tes6ng
is
required
• Move
onto
next
test!

27. Some
Recommended
Reading
/
Resources
[Books]
A
selec6on
of
relevant
books,
blogs
&
websites…
[Blog,
ArAcles,
Reports,
Events
&
Templates]
[Blog
&
Reports]
[Blog,
ArAcles
&
Reports]
[Reports
&
Actual
Test
Results]

28. Thank
You!
au.linkedin.com/in/nraward
slideshare.net/nraward
twiper.com/thecroguy