Bruce Ingraham (Ingraham Consulting) gave a talk on Satisfaction and Loyalty at the SF Data Mining event: http://www.meetup.com/Data-Mining/events/68283282/

1. Using Customer Satisfaction to
Model Loyalty
Bruce Ingraham
Ingraham Consulting
SF Data Mining
Predicting Consumer Behavior
June 19, 2012
San Francisco, CA
1

2. Overview
• Background
• Theoretical framework
• Method
• Results
• Discussion
• Questions
2

3. Background
• Financial services company “Golden Investments”
• Over 30,000,000 customers
• Client had a very good model to predict risk of
defection based on customer attributes and
transaction data
• Client also had monthly customer satisfaction
surveys
• Does satisfaction data provide any additional
information about defections?
3

4. Theoretical Framework
• Mittal, V., & Kamakura, W. A. (2001).
Satisfaction, repurchase intent, and
repurchase behavior: Investigating the
moderating effect of customer characteristics.
Journal of Marketing Research, 38, 131-142.
• Three components in model
– Differential satisfaction thresholds
– Response bias
– Non-linear functional form
4

5. Theoretical Framework
• Differential satisfaction thresholds
– Customers have different “pain thresholds” or
tolerance levels with respect to the decision to
defect
– For example, new customers may be sensitive to
customer service issues
– Satisfaction thresholds vary systematically with
customer characteristics
5

6. ILLUSTRATION: Same Satisfaction Thresholds
If two groups of customers which differ in one characteristic, e.g. tenure, have the same
thresholds, their response graphs have nearly identical lines, reflecting similar retention behavior
for the same satisfaction level.
6

7. ILLUSTRATION: Differential Satisfaction Thresholds
If the groups have different thresholds, their response graphs have parallel lines, since given the
same rating, customers with lower thresholds are more likely to remain customers.
7

8. Theoretical Framework
• Response bias
– Likert-scale satisfaction ratings are error-prone
measures of unobservable (latent) true
satisfaction
– Harsh-raters and easy-raters
– Response bias varies systematically with customer
characteristics
8

9. ILLUSTRATION: Response Bias
If two groups of customers which differ in one characteristic have differential thresholds and
the same response bias, then their response graphs look like the previous chart—parallel lines.
However, if the response bias differs systematically between groups, the additional variation in
retention appears as unequal slopes.
9

10. Theoretical Framework
• Non-linear functional form
– Likert-scale data is ordinal categorical at best
– The relationship between satisfaction and percent
defecting is not interval data
– For example, the difference in percent defecting
between a satisfaction rating of 1 and a rating of 2
may be different from that between a 4 and a 5
10

11. ILLUSTRATION: Nonlinearity
When using a Likert scale to measure satisfaction, the assumption is often made that the difference
in response between rating levels is constant, i.e. the functional form is linear. When this
assumption is true, the response graph is a straight line, and the ratings can be treated as interval
data in modeling. When the assumption is false, the ratings need to be modeled as ordinal
categorical data.
11

12. Theoretical Framework
• Binary probit model
• Main effects parameters γi capture the
different thresholds for each customer group
• Interaction parameters δj capture the different
response biases for each customer group by
rating
12

13. BACKGROUND: Probit Models
Probit models are used to model a binary response variable, e.g. retained/defected, when it is
assumed that the response depends an individual’s threshold being surpassed by an input
variable, such as satisfaction. The threshold is assumed to vary among individuals, and to be
normally distributed within the population.
13

14. BACKGROUND: Probit Models
Inthis model, the input variable is the satisfaction rating. Individual variation is modeled by
characteristics, e.g.loyaltyscore, and interactions with the rating. The resulting threshold value
gives the customer’s position in the normal distribution, e.g. z-score, and the CDF is used to find
the corresponding probability of retention.
Probability of
retention = .84
Threshold = 1
14

15. Method
• Data
– Responses from 12 monthly surveys were
combined to reach a sample of N=9,105
– The survey question of interest was worded
“How likely are you to continue doing business
with Golden Investments?”
– Ratings were on a 5-point Likert scale, where 1
indicated highly unlikely, and 5 indicated highly
likely
15

16. Method
• Data
– Using the predictive defection model, each
customer was assigned to a defection segment
– Risk % of population
• Very high 10%
• High 10%
• Moderate 15%
• Low 15%
• Very low 50%
16

17. Method
• The dependent variable was if the customer
had defected within six months of the survey
response
• Model estimation
– Parameter estimation and hypothesis testing for
the probit model were carried out using SAS PROC
PROBIT
– The reference levels
• rating: 5
• risk: very low
17

18. Method
• Test for addition information about defection
– χ2 goodness-of-fit test to compare actual and
predicted frequencies for the defection model and
the probit model
18

19. Results
• Main effects parameters γi
– Intercept 2.9 (>.001)
– Rating γp
• 1 -1.3 >.001
• 2 -0.9 >.001
• 3 -0.05 >.001
• 4 -0.2 .066
• 5 reference
19

20. Results
• Main effects
– Risk γp
• VH -1.5 >.001
• H -1.1 >.001
• M -0.8 >.001
• L -0.4 .01
• VL reference
20

21. EXAMPLE: Find the retention rate forrisk=High, rating=2.
We start with the intercept, 2.9, which tells us the position of the reference
group in the normal distribution. The reference group,risk= Very Low, rating =
5, has a retention rate of 99.8%.
99.8%
2.9
21

22. Next, the estimate forrisk= High is -1.11. Add this to 2.97 to get the new
position, 1.86. This reduces the retention rate to 96.8%.
96.8%
1.86
22

23. Finally, the estimate for rating = 2 is -0.89. Add this to 1.86 to get the new
position, 0.97. This reduces the retention rate to 83.1%, which is the answer!
If significant interactions between risk segment &rating had been found, they
would be added in also.
83.1%
0.97
23

24. Start at 99.8
risk moves to 96.8
rating moves to 83.1
24

25. Results
• Interactions
– None of the interaction estimates were
statistically significant
– Suspect Type II error: n too small in some cells
relative to the variance, resulting in large s.e.
25

26. Results
• χ2goodness-of-fit tests performed for the defection model
and for the probit model. The expected values for the risk
model were obtained by averaging the risk scores for all of
the survey customers in the segment, and then multiplying
by the segment frequency. The expected values for the
probit models were calculated similarly, using probabilities
estimated by the model.
• The null hypotheses was rejected for the risk model
(p=0.035). The large residuals which indicate lack of fit
belonged to the Very High and Low segments.
• The null hypotheses was not rejected for the probit model
(p=.99). The residuals were very small for all segments, and
the fit was nearly perfect.
26

27. Discussion
• Differential satisfaction thresholds
– Statistically significant main effects are evidence for
this claim
• Response bias
– Statistically non-significant interaction terms do not
support this claim
– May be Type II error due to small n
• Non-linear functional form
– Statistically non-significant estimate for rating of 4 is
evidence for this claim
27

28. Discussion
• Additional information about defection
– χ2 tests are evidence that the survey data provides
additional information about defection
• Issues
– Non-response bias and propensity to respond. Are
unhappy customers less likely to respond?
– Overfitting
– Predicted risk as independent variable
• Business application is to target retention
programs to highest-risk segments
28

29. Questions?
29

30. Thanks for coming!
30