1.
Emi Kwon
UX Lead @MetLife Japan
Why Data-Driven Design
for User Experience?

2.
Why Data-Driven Design for User Experience?
Data-Driven Design for User Experience

3.
Data-Driven Design for User Experience
The decision fatigue
What if I was wearing a face mask?
What if I was not wearing a face mask?
What if________?
We make some ###
decisions every day
…and a lot of it is based
on ### hypotheses

4.
Hypothesis >> Causal data >> Optimized decision (design!)
(a) Hypothesis + (b) causal data + (c) decision =
change with an impact (with less decision fatigue)

5.
Data-Driven Design for User Experience
Data ecosystem — Data lake, data warehouse…what does it mean for UX?
Small data and big data — The opportunities and pitfalls
Research method basics — Qualitative, quantitative or triangulated
User analytics and A/B testing
What about COVID-19 and remote usability testing?
Q&A and Discussion
Contents

6.
Unstructured data
Semi-structured
data
Structured
data
Data that has no inherent structure
and is usually stored as different
types of files
E.g. Text documents, images, videos
Textual data files with an apparent
pattern, enabling analysis
E.g. JSON and XML files
Data having a defined data model,
format and structure
E.g. Database
Data structure
Data-Driven Design for User Experience

7.
Data lake vs data warehouse
Data-Driven Design for User Experience

8.
Data lake vs data warehouse + data mart
Data-Driven Design for User Experience

9.
Not Customer Facing Customer Facing
Sales App 1
Sales
Engagement tool
TTL admin tool
Sales App 2
New Wave pilot
ATS
Post-Transaction
tool
Sales Website
New Wave for
Customer
Transaction
App
Customer App2
Smart Simulator
Marketing Website
Video Chat Chatbot
Customer
Micro-site
Customer App 1
Sales
Customer Sales Admin
New Wave App
New Wave Web
Pilot
Existing Proposed Active Inactive
SimplePath
PIC Kim
PIC John
PIC James
PIC Wendy
PIC Jane PIC Mark
PIC Bryan
PIC Jane PIC Jack
PIC Henry
PIC Tom PIC Anthony
PIC TBC PIC Wendy
Mapping out your data ecosystem can be like…
Data-Driven Design for User Experience
PIC TBC
PIC Henry
PIC Jack
PIC TBC
PIC Annie PIC TBC

10.
Gender Age Group
Per Page
Use
Time Per
Page
# Logins
# Login
Errors
#
Registrat’n
# Reg
Completed
# Reg
Bailed
Avg Reg
Time
Sales App 1
Now: ✖
Future: ⃝
Now: ✖
Future: ⃝
⃝ ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ ＊
Sale App 2 △ △ ⃝ ⃝ ⃝ ⃝ ✖ ✖ ✖ ✖
Sales App 3 △ △ ⃝ ⃝ ⃝ ⃝ ? ? ? ?
SmartPath ? ? ⃝ ⃝ ? ? ✖ ✖ ✖ ✖
NewWave
App 1
? ? ⃝ ⃝ ? ? ✖ ✖ ✖ ✖
NewWave
App 2
△ △ ⃝ ⃝ ? ? ? ? ? ?
NewWave
Web 1
△ △ ⃝ ⃝ ✖ ✖ ✖ ✖ ✖ ✖
NewWave
Web 2
? ? ⃝ ⃝ ✖ ✖ ✖ ✖ ✖ ✖
Admin tool 1 ✖ ✖ ⃝ ⃝ ✖ ✖ ✖ ✖ ✖ ✖
Admin tool 2 ✖ ✖ ⃝ ⃝ ✖ ✖ ✖ ✖ ✖ ✖
△ : should, but no defined requirement as yet ＊ : for individual pages or for registration as a whole?
Mapping out your data attributes can be like…
Data-Driven Design for User Experience

11.
Q: What do you want to get from doing data-driven design? ✨
Data-Driven Design for User Experience
1) To understand my users better
2) Data is the new oil; no way I can do design without being data-driven
3) To add something data-driven to my portfolio/ work experience
4) Not sure yet; will see what I can learn and go from there
5) Others (Feel free to write your answer in the chat 🙂)

12.
Data-Driven Design for User Experience
Data ecosystem — Data lake, data warehouse…what does it mean for UX?
Small data and big data — The opportunities and pitfalls
Research method basics — Qualitative, quantitative or triangulated
User analytics and A/B testing
What about COVID-19 and remote usability testing?
Q&A and Discussion
Contents

13.
Data-Driven Design for User Experience
The data fatigue..
So much data is generated.
So much data is recorded.
So much data is accessible.
In this data lies many
many stories…and among
them, many many stories
of users…
Analysis paralysis..

14.
Data-Driven Design for User Experience
https://www.youtube.com/watch?v=TzxmjbL-i4Y
Big Data!!!

15.
Goals, Location, Data structure, Data
preparation, Longevity, Reproducibility…
Volume, Velocity, Variety
+ Value and Veracity
3 ways big data is different from small data
Data-Driven Design for User Experience

16.
Big data and small data… what does it mean for UX design?
Data-Driven Design for User Experience
https://www.kaushik.net/avinash/best-web-analytics-tools-quantitative-qualitative/
Call center log
Call center log , Sales and revenue data

17.
Data vs Insight
Data-Driven Design for User Experience
Insight
Organized, structured, categorized,
useful, condensed, calculated
Individual facts, figures, signals,
measurements
+ Context
+ Meaning
+ Perspective
Data
Information
Knowledge
Understanding, integration, applied,
reflected upon, actionable, accumulated,
principles, patterns, decision-making
process
Idea, learning, notion, concept,
synthesized, compared, thought-
out, discussed
Decision
risk
Value
Volume,
velocity
&
variety

18.
Data
lake
>>
Data
warehouse
>>
> UX Data mart >> Data-driven design decisions
The way we (would like to) fertilize our design…

19.
Call center log
Call center log
What if only big data exists?
Data-Driven Design for User Experience
Usage data
Only big data ? Call center log

20.
Call center log
Call center log
Data-Driven Design for User Experience
Behavioral insights
Only small data ?
What if only small data exists?

21.
What if only small or big data exists?
Data-Driven Design for User Experience
You believe you know what users want… Without knowing what their critical contexts are….

22.
Data-Driven Design for User Experience
#multiplicity
“The quest for a "single source of the truth" on the web is futile…
…the quest for a single tool/source to answer all your questions will ensure that your
business will end up in a ditch, and additionally ensure that your career (from the
Analyst to the web CMO) will be short-lived… |
https://www.kaushik.net/avinash/big-data-imperative-driving-big-action/

23.
Multiplicity of data leads to good perspective-taking
Data-Driven Design for User Experience
You know the real risk lies in somewhere not so visible… So you will continue to look out for more evidence

24.
Q: Is big data important in doing your work
(design, production, development…)? ✨
Data-Driven Design for User Experience
1) A lot; using big data can make or break a deal in my line of work
2) To some extent; it’s good to have big data around
3) Hm…. might be, but no specific use case yet
4) I don’t care for big data
5) Never a deal-breaker

25.
Data-Driven Design for User Experience
Data ecosystem — Data lake, data warehouse…what does it mean for UX?
Big data and small data — The opportunities and pitfalls
Research method basics — Qualitative, quantitative or triangulated
User analytics and A/B testing
What about COVID-19 and remote usability testing?
Q&A and Discussion
Contents

26.
26
User research & test methods
Data-Driven Design for User Experience
Why What

27.
Data-Driven Design for User Experience
Qualitative research_Interview flow
• Sampling & Participant recruiting
• Drafting questionnaire
• Data collection • Data analysis
• Survey report
• Cross-functional action plan
• Development
• Conduct interview
• Team de-brief

28.
Qualitative analysis tools
Data-Driven Design for User Experience
Tag and organize your interview notes https://www.aureliuslab.com/ https://evernote.com/

29.
29
Data-Driven Design for User Experience
Qualitative analysis_Thematic analysis & Word cloud mapping

30.
• Samping & Participant recruiting
• Drafting questionnaire
• Data collection • Data analysis
• Survey report
• Report share-out
• Building survey form
• Distribution of the survey
30
• Cross-functional action plan
• Development
Survey 2.0 https://www.qualtrics.com/
Quantitative research_Survey flow
Data-Driven Design for User Experience
https://www.jotform.com/table-templates/rainbow-sheet
Survey 1.0 https://www.surveymonkey.com/

31.
31
Data-Driven Design for User Experience
Quantitative analysis_Describe, Relate, Regression, PivotTable
https://www.qualtrics.com/support/stats-iq/analyses/describe-data/

32.
32
Quantitative analysis tools
Data-Driven Design for User Experience

33.
Data-Driven Design for User Experience
Data ecosystem — Data lake, data warehouse…what does it mean for UX?
Small data and big data — The opportunities and pitfalls
Research method basics — Qualitative, quantitative or triangulated
User analytics and A/B testing
What about COVID-19 and remote usability testing?
Q&A and Discussion
Contents

34.
User analytics…what about it?
Data-Driven Design for User Experience
Dimension
• Content information
• Conversion funnels
• Navigation paths
• Visitor details
Strategy
• Acquisition
• Adoption
• Engagement
Metrics
• # of users and sessions
• Average session duration
• Average pages per session
• New/ Returning visitors
• Bounce Rate
• …
34
Dimensions
Metrics
Flow visualization
Adobe Analytics dashboard

35.
User analytics…what can they do?
Data-Driven Design for User Experience
• Real-time events
• Heatmaps
• Session replay videos
• A/B testing

36.
Data-Driven Design for User Experience
Building it together and socializing your UX data

37.
A/B testing
Data-Driven Design for User Experience
What do to
• Getting a testing tool
• Define your UX success metrics
• Define hypothesis to be tested
• Define variables and learning
• Create the variables
• Measure and activate
• Analyze the results
• Document learning & Evangelize
• Ideate & Repeat
https://www.optimizely.com/
https://www.convertize.com/ab-test-significance/
https://business.adobe.com/products/target/adobe-target.html
Control Treatment

38.
https://www.optimizely.com/insights/blog/how-obama-raised-60-million-by-running-a-simple-experiment/
How Obama raised $60 million
by running a simple A/B test
6 Media + 4 Button = 24 combinations.. 40.6% lift in sign-up rate

39.
Data-Driven Design for User Experience
Numbers can inform you of
what is happening,
but they won’t tell you
why something is happening.
User analytics and A/B testing_The pitfalls
#multiplicity
Now you need to dig deeper with qualitative data.

40.
Data-Driven Design for User Experience
Data ecosystem — Data lake, data warehouse…what does it mean for UX?
Small data and big data — The opportunities and pitfalls
Research method basics — Qualitative, quantitative or triangulated
User analytics and A/B testing
What about COVID-19 and remote usability testing?
Q&A and Discussion
Contents The why

41.
What about COVID-19 and remote usability testing?
https://bootcamp.uxdesign.cc/remote-usability-test-heres-all-the-tools-you-need-403717336552

42.
Data-Driven Design for User Experience
Remote usability testing
• Sampling & Participant recruiting
• Drafting questionnaire
• Data collection • Data analysis
• Survey report
• Building survey form
• Distribution of the survey
• Cross-functional action plan
• Development
• Conduct interview
• Team de-brief

43.
Mon Tue
8/10
8/17
8/24
8/31
9/7
9/14
8/11
8/18
8/25
9/1
9/8
9/15
8/12
8/19
8/26
9/2 Emi OOO
9/9
9/16
8/13
8/20
8/27 Emi OOO
9/3
9/10
9/17
8/14
8/21
8/28
9/4
9/11
9/18
Tue Wed Thu Fri
Test scenario (E)
Eye-tracking hands-on (S)
Eye-tracking concept study (W)
Eye-tracking test (E, J, H)
Data-driven UX
monthly call
w/ Biz, Data Science
Test scenario (E, H) Test mockup (H) & Compliance Form (E)
Executive summary (E, M)
Test analysis
BA sharing (J)
Data collection (H)
Compliance Form (E)
User test (E, W)
Data collection (E, W)
Thematic analysis (E) Test analysis
BA sharing TBC (E)
Demo drill (E,S)
Projects overview
Sales App 1 (Emi, Tom)
An app description is an app store optimized
product definition. It greatly influences your
product's success. There are three components
of an app's 'definition': its Name, its Description
in the marketplace, and Screenshots.
Sales App 2 (James, Hannah)
An app description is an app store optimized
product definition. It greatly influences your
product's success. Continuous touchpoint:
whenever you need, you can get it done, fast
and easy.
NewWave Website pilot (Michael, Wendy)
An app description is an app store optimized
product definition. It greatly influences your
product's success. Continuous touchpoint:
whenever you need, you can get it done, fast
and easy.
Test scenario TBC
Eye-tracking
demo w/ DevOps
Data-driven design calendar
Data-Driven Design for User Experience

44.
But don’t be a user test ninja…
Every day A/B test
with 6 design variants plus…
Try all these methods this and next month…
No, you can’t! It will just show
your lack of confidence!
https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

45.
Dunning-Kruger Effect
Time will come
when you hit
this mark…
Every day A/B test
with 6 design variants plus…
ry all these methods this and next month…
https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

46.
Hypothesis >> Causal data >> Optimized decision (design!)
(a) Hypothesis + (b) causal data + (c) decision =
change with an impact (with less decision fatigue)

47.
Data-Driven Design for User Experience
Make informed decisions with causal data
What if I was wearing a face mask?
What if I was not wearing a face mask?
What if________?
We make some ###
decisions every day
…and a lot of it is based
on ### hypotheses

48.
Data-Driven Design for User Experience
Q&A and Discussion
Data-Driven Design for User Experience