Recent technological advances, a dynamic competitive landscape, and an evolving regulatory environment have led to a period of rapid innovation for many insurance providers. Here, we’ll explore how data scientists may use randomized experiments to rigorously assess the causal impact of innovations on business outcomes. Particular emphasis will be placed on experimentation in “offline” channels, with some of the challenges and mitigation strategies highlighted.

1. Supporting Innovation in Insurance
with Randomized Experimentation
Matt Best
Senior Data Scientist
Allstate Insurance Company
DOMINO DATA SCIENCE POP-UP CHICAGO

2. Examples from Ronny Kohavi’s “Introduction to A/B Testing” at KDD ’17 and
Blake and Coey 2014: Why Marketplace Experimentation is Harder than it Seems

3. But, it’s difficult to accurately forecast the impact of an innovation on
customer experience

4. Randomized experiments are an effective
method to learn the impact of innovation

5. Randomized experiments are an effective
method to learn the impact of innovation
Random
sampling

6. Randomized experiments are an effective
method to learn the impact of innovation
• Randomized experiments are also sometimes referred to as
randomized controlled trials, A/B/n tests, bucket tests, and field
experiments depending upon discipline
Treatment Group
Control Group
Random
sampling
Random
assignment

7. CUSTOMERS CUSTOMERS
Web Mobile Service Rep AgencyWeb Mobile

8. Before an experiment, consider the
importance of statistical power
How much data is needed
to assess an innovation’s
impact?
How large does the impact
need to be for it to be
detectable with a fixed
quantity of data?
Ideally,
we’d ask …
In practice,
operational constraints often
shift the question to …

9. Note: All axes units are arbitrary to keep proprietary information confidential

10. Implied MDE
Fixed sample size
Note: All axes units are arbitrary to keep proprietary information confidential

11. Optimistic estimate
of impact
Note: All axes units are arbitrary to keep proprietary information confidential
Implied MDE
Fixed sample size

12. Optimistic estimate
of impact
New sample size
Note: All axes units are arbitrary to keep proprietary information confidential
Implied MDE
Fixed sample size

13. A power analysis saved us from running a test
with almost no chance of success!
Optimistic estimate
of impact
New sample size
Note: All axes units are arbitrary to keep proprietary information confidential
Implied MDE
Fixed sample size

14. Key takeaways before the experiment begins
• Lessons learned:
• Need to be able to rapidly iterate on power/sample analysis and experimental
design as operational constraints are identified
• Observations are rarely independent and identically distributed; be explicit
about sources of variability
• Technological solutions:
• Using a knowledge management platform has enabled us to track the
evolution of assumptions through the design process
• Developed python package to verbosely describe and simulate progress
through process flows

15. After an experiment, consider how cognitive
biases influence decision making
Treatment Group
Control Group

16. Treatment Group
Control Group
Confirmation bias
We look for and more strongly weigh information that confirms
what we already believe
Look again…my
hypothesis must
be true!

17. Treatment Group
Control Group
Hindsight bias
After we see results, we tend to overestimate how well we
would have predicted (or did predict) those results all along
That result was
obvious! Why
run a test?

18. How to benefit from hindsight, prospectively?
• Pre-mortem: “Imagine your experiment has spectacularly failed –
write the story of that failure.”
• Pre-register: “What would you do if we observe a
{positive|negative|null} result?”
• Good decision ‘hygiene’ helps reveal critical risks, assumptions, and
disagreements early on… while we can still do something about it!

19. Summary and Closing Thoughts
Randomized experimentation is a powerful tool data scientists may
leverage to create value.
Though challenging, insurance firms may benefit from wider
adoption of the methodology, even in situations where it’s
operationally challenging.
Data scientists can enable experimentation by driving forward both
technological and cultural solutions.

20. Thanks for your attention!
XD Team
• Anthony Pham
• Andrew Mehrmann
• Matthew McAuley
• Melissa Alvarado
• Nicholas Syring (intern)
BehavioralSight
• Linnea Gandhi
Allstate - D3
• Xiaoyan Anderson
• Neal Coleman
• Tony Eberle
• Florent Buisson
• Jason Khan
Domino
• Anna Anisin
• Jeremy Mason

21. Data and Analytics at Allstate: Our Centralized Organization
Managing and governing
data
Developing analytics
solutions
Effectively delivering solutions
through technology
250 data and analytics
experts
Who We Are
We have experts across five locations:
Silicon Valley, CA; Seattle, WA; Northbrook, IL;
Chicago, IL; Belfast, Northern Ireland
Data and analytics is embedded in
everything we do. Each day, Allstate uses
analytic models to create millions of
targeted digital media impressions, process
tens of thousands of claims, produce tens
of thousands of quotes, and predict
thousands of decision making actions
across the corporation.
Where We Work What We Do

22. Join Us for a Tour of the Allstate Office
Sign up before noon at:
Registration desk or Allstate booth
Tuesday, November 14
3:30 - 4:00 pm
Meet at the Allstate booth