5. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
..TO MEASURE THE QUALITY OF UX
H O W TO U S E T H E DATA
.. TO INFORM YOUR DESIGN ..TO DESIGN AI / MACHINE
LEARNING EXPERIENCES
User-
Research
Strategy
Solution
THIS TALK
.. TO VALIDATE YOUR DESIGN
7. 2017 Noémie PRIN - IAE Grenoble
C A T C H I N G B I G G E R W A V E S
8. 2019 Noémie PRIN
“What has been
affirmed without proof
can also be denied
without proof.”
― Euclid
‣ Make your design valuable
Everything that can be measured and can be improved
We can only optimise complexity ex-post
‣ Objectivise your design decision.
Gain credibility and relevancy toward your stakeholders
‣ ROI calculation
Where do we have to prioritize our effort?
‣ How do we know our work is working?
‣ Team cohesion
W H Y U S I N G U X M E T R I C S
9. 2019 Noémie PRIN
4 .
D E C I D E
3 .
T R A C K P R O G R E S S
& A N A LY S E
1 .
D E F I N E
Change your design
• Reduce cognitive /
motor / visual load
• Automation
• Connect Apps, leverage
existing tools
• Improve Onboarding
Sunset features
Conduct qualitative
study
for further investigation
Run new experiments
A/B test, fake doors..
ROI
Impact vs Design effort
Observe
Benchmark
Identify Correlation
Text mining
Metric tracker
UX Scorecard
Calculate Experience
Quality Score
Communicate
Dashboard
Data visualisation
North Star KPI
Performance
Conversion..
3 UX metrics
• Happiness
• Engagement
• Adoption
(Onboarding)
• Retention
• Task Success
2 .
C A P T U R E DATA
Set up the technology
With analytics package
Event monitoring tools
Data science Tools,
Technical “by hand”
Custom code, Excel exports
Create data via User Research
Quantitative (at scale)
Qualitative (expansive)
Explore from other data
sources (Insights from Customer
support, open data, Google
trends..)
11. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
a great UX metric…
...makes you look at all the other metrics and say
“none of those other numbers matter if we don’t get this right first.”
13. 2019 Noémie PRIN
Hi, I’m Noemie UX designer
Cool, I’m Joe, Responsible of…..
How can I help?
So...
14. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
WHAT ARE THE COMPANY KPIS
AND CORE BUSINESS GOALS ?
S TA R T W I T H A Q U E S T I O N
15. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
Scale the business
“How do we manage significantly
more customers with no additional
relationship managers?”
M Y E N T E R P R I S E C A S E
G E N E R AT E VA LU E T H R O U G H O P E R AT I O N A L E F F I C I E N C Y
Key UX metrics
Engagement
Task success
Business metrics to correlate with
Sales user performance
! Limit your set of UX metrics
! Relate it to company KPIs
! Measure the usage of your
product by a person.
! Specify it to features that
deliver value to your user.
16. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
M E A S U R I N G E N G A G E M E N T
The frequency, intensity, or depth of interaction between user
and product, feature, or service over a given period of time.
I S D I F F E R E N T T H A N …
SATISFACTION
MOTIVATION
17. 2019 Noémie PRIN
1. Start tracking product engagement
activities that will inform your score
2. Weigh each product engagement
event based on how much it should
affect your score
3. Apply the product engagement scores
to make them actionable
E N G A G E M E N T S C O R E
Events ex:
ompleted form x
Comment field in X record page
Changed opportunity stage to stage 4
1.Rank your Users
2.Calculate overall engagement score
3.Track score progress
18. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
S P E C I F I C E N G A G E M E N T M E T R I C
! Actionable
! Understandable
! Easy to communicate
! Avoid vanity metrics
Unhelpful Vanity Good Better Awesome
Start with
• “number of...” [ # ]
• “average number of...” [avg ]
• “percent of...”
19. 2019 Noémie PRIN
Set a goal (flow rather than stock)
Set a time span
Think about how to collect the data
ENGAGEMENT METRIC PLAN ENGAGEMENT METRIC TRACKER
21. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
TA S K S U C C E S S
Task success rate
Time-on-task
Salesforce ex:
Time to complete a quote
Time to complete a account
form
Form completion rate
Allows you to..
!Improve information architecture.
!Identify problem areas and fix them.
!Identify successful areas and don’t fix them.
Time to perform a task
Time taken on the first attempt
Time correcting errors
22. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
TA S K S U C C E S S
Lostness
23. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
Usability tests User behaviour analytics packages
Unmoderated at scale (UserZoom)
Moderated
TA S K S U C C E S S TO O L S
25. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
AV O I D A S K I N G U S E R S
● Selection Bias
● Hawthorne Effect
● Intrusive action (vs passive)
! Directly measure task success
performance and behavior rather
than opinion and attitude
SURVEYS
26. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
H A P P I N E S S M E T R I C E X A M P L E
Rage Clicks
measured in JavaScript by looking for
more than 2 clicks within the same 10
pixels in a short period of time.
28. 2019 Noémie PRIN
A N A L Y S E
Correlation
This appears at the same time
Clusters
Causality
An event causes the other
Linear Regression
DESCRIBE
EXPLAIN
- Data analysts can help you..
T R A C K
- Calculate Experience quality Score
- UX Scorecard
- Dashboard
- Metric Tracker
- Benchmark
- Explore other metrics
Study the behaviour of your users through the data traces they
leave behind them
! What can impact conversion rate?
! What behaviors impact performance?
29. 2019 Noémie PRIN
QUANTITATIVE = WHAT
QUALITATIVE = WHY
C O M B I N E W I T H Q U A L I T A T I V E
Don't solve symptoms. Seek to identify the underlying problem.
30. 2019 Noémie PRIN
U N E D E M A G A Z I N E W E B
H O W TO C O M M U N I C AT E
Dashboard UX Scorecard
Metrics tracker
Excel
Slides
Data visualisation
Google Data Studio
Tableau
D3.js
RAW Graphs
Kissmetrics
Segment
Hotjar
Mixpanel
Einstein analytics
32. 2019 Noémie PRIN
Sunset features
Review (or kill) any features that few people are
using very little
Change your design
• Reduce cognitive / motor / visual load
• Automation
• Connect Apps, leverage existing tools
• Improve Onboarding
Conduct qualitative study
for further investigation
Run new experiments
A/B test, fake doors..
ROI
Impact vs Design effort
34. 2019 Noémie PRIN
Increased user
satisfaction
Increased user
adoption
Increased user
productivity
Decreased training
and support costs
U X VA LU E C A LC U L AT I O N
(time saved)
x (employee cost)
x (employees)
e.g. 2 hours/week x €15/hr x
200 employees
= €6000/week
Better data capture
and fewer user errors
(number of errors)
x (avg. fix time)
x (employee cost)
x (number of employees)
e.g. 2 errors/month x 0.83hrs x €16/hr x
500 employees
= €1,328 per month
(number of training hours saved)
x [(cost of trainer)
+ (cost of employee)]
e.g. 4 hours x 200 employees x (€30/
hr + €15/hr)
= €3600
35. 2019 Noémie PRIN Source : Intercom
Business Value €
or Improvement rate %
Effort (hours)
Time to design
Time to implement
Time to test
Design improvement A
Design improvement B
Compare with the UX team effort
36. 2019 Noémie PRIN
“You don’t have to wait to make your ROI case.
Decisions based on estimates are better than decisions made without
any calculations at all.”
– Susan Weinschenk, Chief of UX Strategy at Human Factors International
