Validation and hypothesis based product management by Abdallah Al-Khalidi

1. S§ Validation and Hypothesis based Product Management

2. Focus • Root cause for problems. • Ditches ideas that irrelevant. Why should you care? Feasible • Solves the real problem. • No wasted efforts. Objectivity • Eliminate personal bias for what causes what. Replicable • Results can be used as facts. • Results are durable. User-Centric • Providing real value to users.

3. The Approach: Step by Step Process (Scientific Method) 1- Observations 2- Hypothesis 3- Experiment 4- Refine 5- Validate Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Refine hypothesis How to measure the effect? Success metrics! FYI: Scientific Method was founded by Ibn Al Haytham around 250 years ago.

4. Observations Observing a trend or a problem.

5. Sources for trends or problems Customer Service SalesOnline Discussions Product Vision

6. Sources for trends or problems

7. Impact Based Prioritization

8. Prioritization Methods Relative Weighting RICE Analysis Based on business metric Based on usage frequency Perfect for new features Perfect for optimizing existing features Considers development effort Considers development effort Reach, Impact, Conﬁdence, Effort

9. Relative Weighting Prioritization Criteria Evaluation Factors Increase Sales; maximize ROI Increase customer trust Establish a competitive advantage Improve productivity Cost of development Value Score

10. Prioritized List Issue Type and ID ROI ICS ECA IP Total Value Value Percent Estimate Cost Percent Priority Add FB sign up Feature 10 Cash on Delivery Bug 3 Feature 7 1 6 1 3 3 1 6 1 6 6 1 1 1 8 1 1 6 6 6 3 11 11 14 11 21 10 20 20 40 55 Add Gmail sign up 3 6 6 6 21 15 Pin Address on map 6 6 3 6 21 35 Bug 5 3 3 3 3 12 25 Feature 8 1 3 6 1 11 40 8% 8% 11% 8% 16% 16% 16% 9% 8% 4% 8% 8% 15% 21% 6% 13% 10% 15% 2 1 1.37 0.54 0.76 2.66 1.23 0.9 0.54

11. RICE Analysis

12. Prioritized List

13. 1- Observations 2- Hypothesis 3- Experiment 4- Reﬁne 5- Validate The Approach: Step by Step Process Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Reﬁne hypothesis How to measure the effect? Success metrics!

14. Hypothesis What is a hypothesis? How to build it? & Good vs Bad one.

15. What is a hypothesis? "If _____[I do this] _____, then _____[this]_____ will happen." If I added gmail login, then number of registered users will increase. If I include Cash on Delivery payment method during checkout, then number of purchases will increase. If I enable customers to pin their location on a map, then number of purchases will increase.

16. Hypotheses Tips Before you make a hypothesis, you have to clearly identify the question you are interested in studying. The question comes first A hypothesis is a statement, not a question Your hypothesis is not the scientific question in your project. The hypothesis is an educated, testable prediction about what will happen. Make it clear A good hypothesis is written in clear and simple language. Keep the variables in mind A good hypothesis defines the variables in easy-to-measure terms, like who the participants are, what changes during the testing. Make sure your hypothesis is “testable” Don't bite off more than you can chew! To prove or disprove your hypothesis, you need to be able to do an experiment and take measurements to see how two things are related. Make sure your hypothesis is a specific statement relating to a single experiment.

17. Good vs bad hypothesis Good Hypothesis Bad Hypothesis Testable Simple Written as a statement Establishes the participants & variables Predicts effect Not testable Not simply explained Written as a questions Doesn’t identify participants & variables Cannot use to predict effect

18. Good vs bad hypothesis Good Hypothesis Bad Hypothesis If I added gmail login, then number of registered users will increase. If I include Cash on Delivery payment method during checkout, then number of purchases will increase. If I enable customers to pin their location on a map, then number of purchases will increase. Would adding more login options increase number of registered users? Cash on delivery payment is requested by users We have 10,000 calls from users to track their orders

19. The point is to prove or disprove a hypothesis Disproving a hypothesis matters as much as proving it. Both leads to form better conclusions about the problem at hand!

20. Sources for trends or problems

22. Experiment How to test hypothesis? What data to look at?

23. Creating a mathematical model What questions do you want to answer? What data do you want to look for? Where the data you want is available? How to use the data available? How to verify the data? Will adding Gmail login improve sign up? How many customers have signed up using Gmail email? Customers data base Find % of customers signed up with Gmail. Gmail customers / all customers = % of gmail customers. Consider removing dummy created accounts

24. Creating a mathematical model What questions do you want to answer? What data do you want to look for? Where the data you want is available? How to use the data available? How to verify the data? Will adding Gmail login improve sign up? How many customers have signed up using Gmail email? Customers data base Find % of customers signed up with Gmail. Gmail customers / all customers = % of gmail customers. Consider removing dummy created accounts 150K 150K / 1.3M = 12%

26. Reﬁne Reﬁne mathematical model, removing bias!

27. Will adding Gmail login improve sign up? How many customers have signed up using Gmail email? Find % of customers signed up with Gmail. Gmail customers / all customers = % of gmail customers. Consider removing dummy created accounts 150K 150K / 1.3M = 12% What questions do you want to answer? What data do you want to look for? Where the data you want is available? How to use the data available? How to verify the data? Creating a mathematical model Customers data base

28. We live in a dynamic world, so always consider things in a timely manner!

29. Creating a mathematical model What questions do you want to answer? What data do you want to look for? Where the data you want is available? How to use the data available? How to verify the data? Will adding Gmail login improve sign up? How many customers have signed up using Gmail email in the past 6 months? Customers data base Find % of customers signed up with Gmail in the past 6 months. Gmail sign ups / all sign ups in the past 6 months = % of gmail customers. Consider removing dummy created accounts 120K 120K / 600K = 25%

30. 1- Observations 2- Hypothesis 3- Experiment 4- Reﬁne 5- Validate The Approach: Step by Step Process Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Reﬁne hypothesis based on test. How to measure the effect? Success metrics!

31. Validate How to measure the effect? Success metrics!

32. What is a success metric? Metrics are a lens into your product’s health and performance.

33. Success Metric

34. Measure the uplift

35. 1- Observations 2- Hypothesis 3- Experiment 4- Reﬁne 5- Validate The Approach: Step by Step Process Questions to answer. Observing a trend or a problem. What is a hypothesis? How to build one? & Good vs Bad hypothesis. How to test hypothesis? What data to look at? Reﬁne hypothesis. How to measure the effect? Success metrics!

36. Thank you Abdallah Al-Khalidi Linkedin: abzkhaldi | Twitter: @abzkhaldi Email: abdallah.khalidi@gmail.com