1. Digital experiments, also known as A/B tests, involve making small changes to digital content and randomly assigning users to different versions to measure the impact of changes. 2. They provide an unambiguous way to measure the impact of changes through randomized controlled experiments, helping teams find problems, save time and money, and improve performance. 3. The presentation provides examples of how to structure different types of digital experiments and tools for implementing experiments on websites, apps, email and more. 4. User research is important to guide the questions being tested to ensure experiments are strategic and focused on key areas of impact.

1. Action Design DC
10 September 2015
Stephen Wendel
Head of Behavioral Science
@sawendel
Digital Experiments
(A/B Tests on Steroids)

2. Hi! I’m Steve.
Head of Behavioral Science
Morningstar

3. Until recently, I was here
Update screen
Personalized financial guidance that transforms
financial lives through behavioral economics

4. Now, I’m at
Leading provider of independent investment advice
serving millions of people everyday, around the world

5. My Job

6. The Path Ahead
1 What are digital experiments?
2 Why they’re awesome
3 How to do them
4 How to be strategic

8. Experiments
in 2 Minutes

9. The product is now awesome!
0
2
4
6
8
10
12
14
16
Before After
Something Really Important

10. It’s often really hard to tell…

11. The Path Ahead
1 What are digital experiments?
2 Why they’re awesome
3 How to do them
4 How to be strategic

12. Reason 1: Unambiguously Clear, Really
Simple Measurement of Impact

13. Know what works (settle arguments and
market the hell of our your success)
Randomized Experiments:
The Gold Standard
Statistical Models w. Controls:
Harder Work, For a Silver Medal

14. Reason 2: Find problems early,
save time and money

15. Hundreds of experiments…
“...If you sign up by
7PM today,
you are eligible to
win a free iPad mini!”
“...If you are one of
the first 100 people
to sign up, you are
eligible to win a free
iPad mini!”

16. Most changes
just don’t do much
(assuming your team is awesome)

17. Our intuition
is
often wrong
(You can’t just
use “the right
version”)

18. Experiments separate the
wheat from the chaff

19. Reason 3: Money (And Success, Fame, Etc.)
300% boost in
managed account enrollments

20. Reason 4: All the cool kids are doing it

21. Digital experiments guide us through

22. The Path Ahead
1 What are digital experiments?
2 Why they’re awesome
3 How to do them
4 How to be strategic

23. Four common terms
Pick one thing to change,
compare two versions
A/B Test

24. Four common terms
Pick one thing to change,
compare three or more versions
A/B/C Test

25. Four common terms
Change more than one thing
With multiple versions of each thing
MVT (Multivariate) test

26. Four common terms
Exploit the “best” performing version,
while devoting some traffic the others
Multi-armed Bandits

27. There are some of the darned nifty tools

28. Websites
Point and Click DOM Manipulation

29. Single Page Applications
Build on open
source libraries

30. Mobile Apps
Use existing mobile tools,
or develop local rendering logic
& use a server-side API

31. Email
Use just about anything.
Other than Outlook.

32. Roll your own

33. Use R (or an online calculator)
To determine how many people you need,
and which side worked

34. The Path Ahead
1 What are digital experiments?
2 Why they’re awesome
3 How to do them
4 How to be strategic

36. What Do You
Want to Learn?

37. User research guides you to the right
questions to ask

38. 1) Does the darned thing work?
Name: an “Impact Test”
Version A: Do Nothing
Version B: Your Communication

39. 2) Can we get anything to work better?
Name: a “Kitchen Sink Test”
Version A: Current Underperforming Version
Version B: Your Best Shot, with everything

40. 3) What fundamentally
resonates with our
users more, X or Y?
Name:
an “Archetype Test”
Version A: Everything is X
Version B: Everything is Y

41. 4) What exactly
drives a result?
Name:
“Isolated Intervention”
(Aka “Microscope Test”)
Version A: A baseline
communication
Version B: The exact
same communication,
with a single small
change

42. 5) Are my tools working?
Name:
“Sanity Test”
(Aka “A/A Test”)
Version A: Something
Version B: The exact
same something

43. Find the leading indictors
that save waiting time.

44. Vet ideas cheaply,
before proving their
impact expensively.

45. Where we’ve been
1 What are digital experiments?
2 Why they’re awesome
3 How to do them
4 How to be strategic

46. Much of the data we use now
is misleading.
0
2
4
6
8
10
12
14
16
Before After
Something Really Important

47. By randomly assigning people,
can we get a clear signal

48. There are some pretty awesome tools that
do this stuff automatically

49. But it’s hell if you don’t know what to ask

50. If you do it right, you win big.
300% boost in
managed account enrollments

51. It all starts by accepting that
we’re probably wrong.