How Yammer uses data to make product decisions

2. Measure or Die
Neil McCarthy
Ken Pascual

3. Justifying features
Execution decisions
Determine success

4. HOW DATA HELPS US PRIORITIZE
Step 1: Justifying Ideas

6. Previous signup flow experiment

9. Clues from the past

10. Brainstorming

11. Idea!

12. Does this align with our goals?
• Vision: Yammer is a primary communication
tool that users use to get work done.
• Goal: Grow the number of users that use the
product.
• Problem: We lose too many potential users in
our signup flow.

13. How important is this?

15. big drop!

16. And HALF of our invites come from the signup flow!

17. Conclusion: This seems important.

18. Why justify?

19. HOW DATA HELPS US DETERMINE
THE SUCCESS OF THE PROJECT
Step 2: Determining Success

20. What’s your hypothesis?

21. What we came up with…
If we move the email confirmation step to the
end of the signup flow,
then more users will make it through the signup
flow and more users will be retained.

22. Secondary objective: Kill the Desktop
app

23. We need another hypothesis

24. “If we remove the prompt to install
the desktop app, then we will lose an
acceptable amount of engagement”.

25. A/B Test Results: Killing the Desktop
app prompt

26. A/B Test Results: Killing the Desktop
app prompt

27. Interpretations
• Hypothesis disproved (in a good way).
• Users find a way to post.

28. Results!

29. A/B Test Results: Moving the email
activation step

30. A/B Test Results: Moving the email
activation step

31. Interpretations
• Users saw the steps in a different order, and
dropped off in similar percentages, BUT
– Less users saw the prompt to “check email”
because it was at the end.
– Thus, less users activated via email.
– Thus, less users had the opportunity to engage.

32. Why we chose this example

33. How do you do this?
• Gotta log data!
• Choose your own metrics
• Data driven vs. data informed
• Build your processes around data
• How much PMs should know about statistics.

34. Q & A

35. Thanks for coming!
• Neil McCarthy, @the_neil
• Ken Pascual, @vorped
• Adam Zadikoff (original PM for this project),
@zadikafka
• Test results from the first signup flow test:
https://medium.com/yammer-
product/13eeb6435a73
• These slides on Slideshare coming soon!