Learning about human behavior is a vital aspect of the experimentation process. You build an idea around what could be the improved digital experience that will influence the buyers to buy your service or products. And then buyers tell you what they think about your idea. At Intuit TurboTax, the pace of learning through experiments has increased in the past few years. This acceleration in learning is attributed to 3 factors: access to self-serve data design principles for rapid decisions technology for rapid prototyping These factors are guided by well-defined innovation frameworks used at Intuit: Customer-Driven Innovation (CDI) Design For Delight (D4D) Rhys Mohun and Ted Chiasson are here to talk about learning at speed at Intuit.

1. Intuit Confidential and Proprietary 1
ACCELERATING
LEARNING

2. Intuit Confidential and Proprietary 2
Ted Chiasson
Head of Experience Design,
Innovation Catalyst
Rhys Mohun
Head of eCommerce &
Experiments
Intuit Canada
Presenters today

3. Intuit Confidential and Proprietary 3
Rapid Prototyping
Technology
Operating with Rigour Innovating with Purpose
Customer-Driven
Experimentation
Self-Serve Data
Decision Principles
Design for Delight (D4D)
WHERE SPEED COMES FROMWHERE SPEED COMES FROM

4. Intuit Confidential and Proprietary 4
What exactly did we accelerate?
IMPACTIMPACT
40% reduction in average deploy time to live audience
31% increased impact from test wins
Almost 2x the number of tests deployed (just one shy!)

5. Intuit Confidential and Proprietary 5
#rigour-works!#rigour-works!
“We recorded more winning
experiments this year than we
even deployed in previous years.”

6. Intuit Confidential and Proprietary 6
INTUIT VALUES BOLDNESS
Our team aims to transform
the way people do their
taxes.

7. Intuit Confidential and Proprietary 7
Rapid Prototyping
Technology

8. Intuit Confidential and Proprietary 8
MEASURABLE
EFFECT IS SMALL
Reaching meaningful
confidence interval with this
much uncertainty will require
many observations.
WHAT IS “BOLDNESS”?

9. Intuit Confidential and Proprietary 9
CHANGE IS LARGE
MEASURABLE
EFFECT IS LARGE
WHAT IS “BOLDNESS”?
There are validity tests you
can run for experiments to
quantify visual changes,
experience changes and
boldness.

10. Intuit Confidential and Proprietary 10
INTUIT VALUES
WINNING TOGETHER
Communicate challenges
early and often, solving
with principles.

11. Intuit Confidential and Proprietary 11
Protect against errors
Critically focus debate
Evolve as you learn
Make allies of stakeholders
Decision Principles
“When presented with {_______},
we choose to {________}.”

12. Intuit Confidential and Proprietary 12
Succeeds,
above range
Test concludes
Succeeds,
within range
Fails,
within range
Fails,
below range
Take action A
Take action B
Decide on the next steps for every
potential outcome:
● decision metric ranges
● observation metric constraints
● floors and ceilings
Take action C
Take action D
Actions can include:
● escalating to production
● iterating into a v2.0 recipe
● re-evaluating the hypothesis
● walking away

13. Intuit Confidential and Proprietary 13
Communicate
early and often
with experiment
stakeholders.

14. Intuit Confidential and Proprietary 14
Example One-Pager Slide

15. 1P | Mobile SKU comparison modal
Problem: Mobile PLU visitors have difficulty comparing features between products and
cannot understand the value between each SKU packaging
LOFA: Mobile visitors will gain choice confidence when they are able to compare between
SKUs using a familiar mobile-friendly interface
Evidence: VoC = customers need to scroll up and down to compare features and
discoverability between assisted products is very low.
Hypothesis: If we added a compare modal for users to the bottom fixed nav then overall
auth rate per traffic will increase because visitors will compare products’ value across
features and have greater confidence in selecting the right SKU among these options.
Link to dataset | Link to VWO experiment | Link to designs
Control (A) Variation 1 (B)
Elevate access
to information
Setup Method Pages Decision Observe
REACT Kojin Split URL Online PLU T2A
take rate,
discoverability
Target segment Start date End date Weeks Dependencies
New mobile visitors 01/01/2020 01/02/2020 4.0 None
Experiment Outcomes Participants Conversions Rate Improvement (ITC)
Metric Index to control
Control 2,000 1,000 50.00% -
Winner 2,000 1,200 60.00% 120 ITC
Strategy Outcomes Audience New auths Plan T2A Precision (ITG)
Estimated Impact -- -- --
+/-
Actual Impact -- -- --

16. Intuit Confidential and Proprietary 16
INTUIT VALUES
DATA MAKING THE
DECISIONS
Test decisions almost make
themselves with the right
metrics measured.

17. Intuit Confidential and Proprietary 17
Agonize over the
success metric.

18. Intuit Confidential and Proprietary 18
Serve
yourself.
Self-Serve Data

19. Intuit Confidential and Proprietary 19
Evaluate complex results.
Composite metrics like RPV
aggregate complex results into a
single metric. Here we declare a
test winner even though overall
auth rate is down.
Big changes!
(Metric owners need to know!)

20. Intuit Confidential and Proprietary 20
Customer-Driven Experimentation
Design for Delight (D4D)

21. Design thinking is a set of tools used in a process
to unlock a mindset for creative problem solving.
Design thinking has a human-centered core.
At Intuit we call it Design for Delight (D4D)

22. Design for Delight (D4D)
DIVERGENCE
Creating choices Making choices
CONVERGENCE

23. Deep Customer Empathy Go Broad to Go Narrow Rapid Experimentation
Initial
framing
Define the
problem
Right
solution
Deliver
Awesome
Experiences
Discover the
PROBLEM
Explore the
SOLUTIONS
Refine the
EXPERIENCE
D4D process

24. Deep Customer Empathy Go Broad to Go Narrow Rapid Experimentation
Initial
framing
Right
solution
Deliver
Awesome
Experiences
Discover the
PROBLEM
Explore the
SOLUTIONS
Refine the
EXPERIENCE
D4D process
Define the
problem

25. EMPATHY IS
FEELING WITH
PEOPLE
Brene Brown
Perspective taking
Staying out of judgement
Recognizing emotion in others
Communicating that emotion

26. Say / Think
Do / Use
Know
Feel
Dream
TACIT
LATENT
OBSERVABLE
EXPLICITInterviews
Follow me homes /
Observation
Generative Sessions
What they... Methodologies Knowledge
Deep
Surface

27. Deep customer empathy
We gain empathy by observing people
where and when they are experiencing
pains or problems.
DISCOVER THE PROBLEM
FOLLOW-ME HOMES

28. Customer problem statement
I am
A narrow description of the customer (not you!) that highlights their motivations,
attributes and/or characteristics
I’m trying to
Desired outcome
But
Problem or barrier
Because
Root cause
Which makes me feel
Emotion
DISCOVER THE PROBLEM

29. Customer problem statement
I am
I’m trying to
But
Because
Which makes me feel
DISCOVER THE PROBLEM
a dogwalker who isn’t terribly comfortable with technology, but I know I
need to track my expenses. Today I do this in an envelope and spreadsheet.
make sure I get every penny of deductions I deserve
I hate the effort of collecting, tracking, organizing and categorizing my receipts
it’s manual, tedious, and time consuming
like I’m wasting time (which is money), and maybe a little
embarrassed, because I’m sure this isn’t the best way to do it.

30. Ideal State
DISCOVER THE PROBLEM
The Ideal State is a description of a future state where an
important customer problem or opportunity has been
solved to such an amazing degree that the outcome
seems almost impossible.

31. Ideal state
DISCOVER THE PROBLEM
In a perfect world
Bold statement of a future state that is borderline
unachievable (perfect outcomes, not specific solutions)
The biggest
benefit to me is:
The improvement in the customer’s life once the
ideal state is achieved.
Which makes me feel
Emotion

32. Ideal state: Toyota
DISCOVER THE PROBLEM
In a perfect world
Bold statement of a future state that is borderline
unachievable (perfect outcomes, not specific solutions)
The biggest
benefit to me is:
The improvement in the customer’s life once the
ideal state is achieved.
Which makes me feel
Emotion
vehicles never break down
reliable car to drive, no worry of break down
safe and confident when I drive

33. Deep Customer Empathy Go Broad to Go Narrow Rapid Experimentation
Initial
framing
Deliver
Awesome
Experiences
Discover the
PROBLEM
Explore the
SOLUTIONS
Refine the
EXPERIENCE
Design for delight process
Define the
problem
Right
solution

34. FROM
Problem State
TO
Ideal State
TELLING YOUR STORY
Now Future
The Journey
Learning plan

35. Intuit Confidential and Proprietary 35
MAP TESTS
TO GOALS
IDENTIFY
LEARNING
GOALS
IDENTIFY
GAPS
PLAN FOR
FUTURE
TESTING
UPDATE
LEARNINGS
FROM EXECUTED
TEST
DEVELOP
A POINT
OF VIEW
Learning plan spans across
multiple iterative tests that
work together to provide POV
Test 1 Test 2 Test 3 Test 4 Test 5
Learning plan
RAPID EXPERIMENTATION

36. Intuit Confidential and Proprietary 36
LOW
RISK
Test 1 Test 2 Test 3 Test 4 Test 5
Learning plan
RAPID EXPERIMENTATION
ACCRUED
CONFIDENCE

37. If we
Describe how the expirement will work
Then
What is the expected outcome or behavior
Measured by
How you will know if it is successful
Hypothesis
RAPID EXPERIMENTATION
Success for
this metric
will be: Minimum criteria for success

38. Rapid experimentation

39. Selecting experiment types
RAPID EXPERIMENTATION
Boldness
How disruptive is
the idea?
How big could it
flop?
Knowledge
How much evidence
do you have?
How compelling is
the evidence?
Audience
How large is the
audience?
How diverse is the
audience?
Effort
How costly would it
be to build?
How costly would it
be to discard?
Variables
How many parts to
the whole?
How many
approaches to each
part?

40. Intuit Confidential and Proprietary 40

41. Intuit Confidential and Proprietary 41
In summary
Gain deep customer empathy
Create a future state
Have a plan & test often iteratively

42. Intuit Confidential and Proprietary 42
Q&A