She'll be sharing use cases, lessons learned and business impacts made using customer experience data and experimentation. She'll also share how combining quantitative data with unstructured quantitative data could provide powerful insights into user behavior that can be leveraged for improving business KPIs. Her team is using AI to support thousands of employees at PayPal and fueling innovation and data driven culture.

1. Customer Experience & Experimentation
J u n e , 2 0 1 9
Kuntal Maya Goradia
1

2. - Experimentation & CX Program Overview
- Use Cases
- Best Practices
- Takeaways
2

3. Data Analytics & Experimentation Across Entire CX Journey
3
Cross Functional CX Org: Marketing Product Operations Risk Compliance

4. Why Do We Experiment?
4

5. Why Experiment? Empower Individual Ideas Over HiPPOs
5

6. Why Experiment? Manage Risk. Test On Small Sample Size
6

7. Why Experiment? Identify Causation
7

8. How Do We Experiment?
8

9. 9
88%
89%
92% 92%
82%
90%
ELECTRONICS FASHION

10. P-Value calculations
Experimentation
Randomized
Traffic/Customer
Split
Experimentation Data Flow
10
PayPal.com
Experimentation
Data
KPI
Type
Proportional
Ratio
Continuous
Chi-Sq
Binomial
T-test
(remove
outliers)
Test Planning Data
• Hypothesis
• Agreement on KPI (1
Primary, 2-3 secondary)
• Minimum Detectable Effect
(min lift) & Minimum
Sample Size Calculations
• Sequential or Fixed
• Holdout allocation
Report observations:
Sample size confirmation,
KPIs lift, 365 Impact

11. Types of CX Experiments
11

12. Types of CX Experiments: Call To Action : Visuals and Copy Edits
12

13. Types of CX Experiments: Page Redesigns
13

14. Types of CX Experiments: Entire Flow Redesign and Form Optimization
14
#1

15. Types of CX Experiments: Personalization
15

16. CX Improvement: Experiment New Ideas
16

17. ©2019 PayPal Inc. Confidential and proprietary.
Account Signup Page
17
Which page has better signup conversion?
Control Treatment

18. ©2019 PayPal Inc. Confidential and proprietary.
Account Signup Page
18
W
inner
Control Treatment

19. ©2019 PayPal Inc. Confidential and proprietary.
Business Homepage Intent Selector
19
Which page has better signup conversion?
Control Treatment

20. ©2019 PayPal Inc. Confidential and proprietary.
Business Homepage Intent Selector
20
W
inner
Control Treatment

21. CX Improvement: Data Insights
21

22. Password Reset: Users Timed Out If Not Completed Reset Within 5 minutes
22
You have 5 minutes to complete the entire Password Recovery Process
You must successfully set your password
before the 5 minutes ends, or you
“timeout”.
A customer may
experience one or
more “Security
Checks”
The timer starts once you
submit your email address.
Password
Reset
Complete
0:00
< 5:00

23. 1 in 13 Consumer Timed-out in PWR
81% Time-outs Between 5-10 mins
23
Data source: Tealeaf
81%
Extended the timer by a few minutes to reset password

24. CX Improvement: Site Feedback
24

25. Used Site Feedback To Identify Address Verification Service Issue
25
Feedback comment:
“I can't even make a account. I try and
sign and put my name and address and
it says check your information and try
again. I checked my information and
nothing's wrong.”
Leveraging NLP to identify the common customer issues
Address verification service rules
needed to improve to match with or
without space in address
“Winter crest Dr” vs
“Wintercrest Dr”

26. Measurement Considerations
26

27. 1) KPI Considerations: Critical To Identify Business Metrics
Conversion Rates Heat Maps &
Session Replay
Voice of the Customer
Revenue
31%
5% 4%
1.2%
0.7%
1.5%
< 1% visitors clicks on
each individual link
< 1% visitors clicks on
each individual link
0.7%
< 1 % visitors clicks on
each individual link
0.1%
3%
0.3%
0.14% 0.08%
27
Customer Experience
Metrics
Business Metrics
Sign-ups Support Cost

28. 2) Apply Segmentation To Identify The Opportunities Within Segments
Identify which segments the test is performing well ( or not) compared to other,
segment the test by customer attributes such as:
• Device Types: Desktop or Mobile
• New or Returning
• Region
• Language
28

29. 3) Plan Measurement Methodology Upfront And Let The Data Reach
Significance To Ensure Quality of Analysis
Greater than
0.05:
Insufficient
evidence
0.01-0.05:
Some
evidence of
statistical
significance
Less than
0.01: Strong
evidence of
statistical
significance
29
P-Value
• Decide between Fixed-sample size vs
Sequential to optimize the time to complete
the test
• An experiment designed to detect a smaller
lift will require fewer observations using a
sequential test.
• An experiment designed to detect a larger lift
will on average require fewer observations
using a fixed-sample test.

30. 4) Monitor Platform Health: Verify Bias Free Random Sampling
Create A/A test
• A/A Test serves up identical experiences for all visitors on a page or experience
Orthogonality vs Interaction Effect Testing
• Regression Analysis for concurrent experiences
Monitor
• Split %: The accuracy of desired split %
• Conversion Rate: The validity of the randomizing algorithm
30

31. Augmented Intelligence
31

32. Leveraging NLP, Augmented Intelligence To Support Internal Customers
32
Stretching analytics teams’ bandwidth via AI chatbot to support internal users

33. 7 Habits of Highly Effective Analysts
33

34. 1. Set Goal: Measurable Business Impacts
34
• Set your goals upfront
• Communicate goals and what you would
deliver at the end of the project
• Goal examples
ü I want to analyze password recovery flow
and identify opportunities to reduce friction
and thus reducing customer contacts to the
call centers.
Expected business impact: xx$ Cost saving
I want to build a report to identify top
paths users take in password recovery flow.

35. 2. Start Small and Then Scale
35
• Plan for scalability but initially focus
on a couple of use cases
• Demonstrate solid methodology and
best practices that can be expanded

36. 3. Don’t Let Perfection Get On The Way Of Progress
36
• Utilize limited data-set that you
do have even if it may not be
complete
• Make the best of what you have
STRIVE FOR
PROGRESS
NOT
PERFECTION

37. 4. Utilize The Eco-System
37
Leverage other tools to
augment insights. Create
seamless integrations with:
• Site feedback
• Client web logs
• Server logs
• Existing Dashboards

38. 5. Empowerment: Self-Service And Training
38
Invest time in empowering cross-
functional teams:
• Democratize data
• Trainings/ FAQs
• Self-service reports
• Alerts
• Utilize Robotics & AI to power
self-service

39. 6. Create a Knowledge Base Library
©2015 PayPal Inc. Confidential and proprietary. 39
• Document the
experiments
• What worked
• What didn’t work
• Create inventory of
analyses
• Insights gathered
• Actions taken

40. 7. Track The Impact And Value Of Analysis
40
• Track value you have added via
analytics and insights
• Partner with business to
implement the actions you
recommend from your analyses

41. Thank You!
kungoradia@paypal.com
41