O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

How to Create Winning A/B Tests through Stronger Research

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Carregando em…3
×

Confira estes a seguir

1 de 44 Anúncio

How to Create Winning A/B Tests through Stronger Research

Baixar para ler offline

This is the accompanying presentation to a webinar on doing research for A/B testing. It describes how online marketers can create better A/B tests by investing in rigorous, useful research to create great testing hypotheses.

The topics covered are
- Analytics (digging into analytics with an eye on useful insights)
- Usability Testing
- On-site surveys
- Visual Analysis
- And finally, pulling all of these together

This is the accompanying presentation to a webinar on doing research for A/B testing. It describes how online marketers can create better A/B tests by investing in rigorous, useful research to create great testing hypotheses.

The topics covered are
- Analytics (digging into analytics with an eye on useful insights)
- Usability Testing
- On-site surveys
- Visual Analysis
- And finally, pulling all of these together

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a How to Create Winning A/B Tests through Stronger Research (20)

Anúncio

Mais de VWO (20)

Mais recentes (20)

Anúncio

How to Create Winning A/B Tests through Stronger Research

  1. 1. How to Create Winning A/B Tests through Stronger Research
  2. 2. View the video of this webinar here
  3. 3. About Siddharth Deswal ● Senior Marketer at VWO ● Passionate about analytics, optimization and storytelling ● Connect on Twitter @SiddharthDeswal
  4. 4. Research - The Scientific Method Problems are usually well defined: Increase sales, signups, leads generated, etc. Neglected part of conversion rate optimization (or just lip-service paid) HiPPOs rule over here (Highest Paid Person’s Opinion) Most users focus on these steps
  5. 5. Why research 1. Only one in seven tests results in a winner 2. Many people don’t know what to test after the initial experiments 3. Tests with results that don’t move the needle
  6. 6. Analytics
  7. 7. Analytics - Top sources
  8. 8. Analytics - Understand entry points
  9. 9. Analytics - Mapping exit pages
  10. 10. Analytics - Key pages
  11. 11. Segment for better insights 1) New visitors who don’t bounce
  12. 12. Segment for better insights 1) New visitors who don’t bounce (continued)
  13. 13. Segment for better insights 2) Visitors who behave similar to those who convert
  14. 14. Segment for better insights 2) Visitors who behave similar to those who convert (continued)
  15. 15. Segment for better insights 3) Finally, analyze the visitors who behaved like converters but did not convert by looking at their top sources, entry points and exit pages
  16. 16. Usability Testing
  17. 17. Usability Testing Putting 5 to 10 users in front of your website or landing pages and observing them can teach you far more than most other forms of research. Caveat: these users have to be representative of your target customers.
  18. 18. Usability Testing - Asking questions “You received a damaged product, try and return it” 1) User searches for ‘Returns Policy’ page and then proceeds to follow the steps listed vs. “You received a damaged product, what do you do now?” 1) User opens ‘Live Chat’ and asks rep what to do, or 2) Takes to Twitter to complain to the brand, or 3) Looks for the ‘Returns’ page, or 4) Something that none of us were able to anticipate
  19. 19. Usability Testing - Asking questions “You received a damaged product, try and return it” vs. Useful when trying to understand roadblocks in a specific process but the question has biased the user towards a particular action “You received a damaged product, what do you do now?” Better when you want to understand how users react to problems without any bias
  20. 20. Usability Testing - Measurement Quantitative measurement 1. Number of clicks to solve the problem 2. Time taken to solve the problem 3. Number of page loads to solve the problem 4. Errors while attempting to solve the problem 5. Number of goals completed while solving the problem
  21. 21. Usability Testing - Measurement Qualitative insights 1. What catches the participant’s attention in the first 5/10/15 seconds after the landing page has loaded 2. How quickly is the headline able to communicate the business’s primary offering 3. How does the participant search for information 4. How convinced is the user with and without trust signs/badges 5. Questions that the user generates while attempting to solve the problem 6. How pleasant the user finds the website 7. Distractions that make participants leave the conversion funnel 8. Points where participants fumble, are confused or have to turn to you for further direction
  22. 22. User Testing - Attensee.com
  23. 23. On-site surveys
  24. 24. On-site surveys
  25. 25. On-site surveys - SaaS Page Question Insights drawn Pricing / Features / Trial-Signup What’s stopping you from signing up with us? Common concerns, objections, anxieties Top exit pages Is there something you were looking for that you couldn’t find on the website? Reasons that users leave, Information that they needed to make a decision but couldn’t find Top landing pages / Homepage How familiar are you with [business offering]? Understand how much convincing/explaining do your visitors need Top landing pages / Homepage Where did you come to know of us? Top marketing channels
  26. 26. On-site surveys - SaaS
  27. 27. On-site surveys - eCommerce Page Question Insights drawn Product page What’s stopping you from adding this product to cart? Common concerns, objections, anxieties or ‘just looking’ Top exit pages (product or category) Is there something you were looking for that you couldn’t find on the website? Reasons that users leave, Information that they needed to make a decision but couldn’t find Top landing pages / Homepage Where did you come to know of us? Top marketing channels Checkout / Cart What’s stopping you from buying this product? Why are visitors not completing purchases in spite of adding to cart
  28. 28. On-site surveys - Exit intent
  29. 29. On-site surveys - Exit intent WAIT Tell Us Why You’re Leaving And Get A 10% Discount Coupon On Your Next Purchase I’m leaving because ● I don’t want to create an account before buying ● I found a better price elsewhere ● I can’t find a coupon code ● I don’t see free-shipping ● I was just browsing ● Other
  30. 30. Visual Analysis
  31. 31. Visual Analysis - Heatmaps
  32. 32. Heatmaps case study - original
  33. 33. Heatmaps case study - Heatmap
  34. 34. Heatmaps case study
  35. 35. Heatmaps case study - Result 25% increase in clicks on app store buttons!
  36. 36. Visual Analysis - Scrollmaps
  37. 37. Hypothesis
  38. 38. Hypothesis Research Hypothesis Image credit http://www.pasosadelantepv.com/evaluation-treatment.html
  39. 39. Hypothesis Framework The Change:Effect model Image credit http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis. html
  40. 40. Hypothesis Framework The Change: Effect model depends on 1. Presumed problem 2. Proposed solution 3. Anticipated result
  41. 41. Hypothesis Framework The Change: Effect model Image credit http://www.marketingexperiments.com/blog/analytics-testing/creating-good-hypothesis. html
  42. 42. Putting it all together 1. Start by creating a segment that shows only those visitors who came from your largest or most important traffic source 2. Get a list of the top landing and exit pages for this traffic source 3. Run user tests where representative participants are asked to solve a problem that takes them through these pages 4. Make note of the key concerns, objections and difficulties expressed by test participants 5. Run on-site surveys on the key pages
  43. 43. Putting it all together 6. Again, make a note of the concerns, objections and reasons given by respondents 7. Run heatmaps and scrollmaps on your key pages to understand visitor behavior in their natural environment 8. In our experience, once you’ve completed the analysis of all the previous steps, the problems on your landing pages and websites automatically jump out, i.e. they become so obvious that everyone is easily able to identify them 9. Create hypotheses that attempt to fix these problems 10. Run your A/B tests!
  44. 44. Thank You!

Notas do Editor

  • Higher chance of success
    A large testing plan
    Tests that have direct impact on business metrics
  • Understand where your visitors are coming from
    Pay special attention to those channels that bring in the highest volume of conversions, and
    The highest percentage conversion rate (these are opportunities that can be scaled further)
  • Where are people landing?
    These pages are like the customer facing staff in a bank or hotel and should be highly optimized
    Landing pages with the high entry AND bounce are A/B testing gold mines
  • Pages with high exit are the holes in your funnel
    Major opportunity for A/B testing as these are places you want to test to plug the leaky bucket
  • Important parts of the user journey as they move through your website
    Help in addressing key visitor anxieties
    Help visitors make their minds up for the final converting action
  • Visitors who just come and bounce inflate the statistics for the landing pages
    While new visitors who don’t bounce are actually the ones you want to optimize deeper parts of your funnel for
    Run the analysis explained in the previous slides to get better insights on visitors who have found a reason to further explore your website
  • Visitors who just come and bounce inflate the statistics for the landing pages
    While new visitors who don’t bounce are actually the ones you want to optimize deeper parts of your funnel for
    Run the analysis explained in the previous slides to get better insights on visitors who have found a reason to further explore your website
  • Create a segment of those visitors who completed a goal action
  • Once you’ve created this segment, have a look at their key metrics and use those to create another segment
    Then, run this new segment called “Behavior similar to converters or buyers” and try and do a deep-dive into their traffic sources, top landing pages and top exit pages
  • Once you’ve created this segment, have a look at their key metrics and use those to create another segment
    Then, run this new segment called “Behavior similar to converters or buyers” and try and do a deep-dive into their traffic sources, top landing pages and top exit pages
  • Ask the participants to vocalize their thoughts while going through this process
    Remember that if participants can’t understand something and have to turn to you, then real visitors don’t have anyone they can turn to and these are points that could lead to confusion and/or bounce
  • On site surveys like WebEngage are a great way to understand what’s happening at the Zero Moment of Truth (ZMOT)
  • For third question “This question helps you understand the sophistication and experience of your visitors. If the majority are familiar with your offering, you need to move them to the Desire stage of the AIDA (Attention Interest Desire Action) funnel”
  • We asked a question to homepage visitors and realized that the majority of our marketing and communication should be around introducing and explaining A/B testing
  • DoDoCase uses BounceExchange exit intent popups to slow you down when leaving
  • Use exit-intent popups to understand why users are leaving at Checkout
  • Heatmaps are the easiest way to quickly visualize where visitors interact most with your website. In the heatmap of our homepage above, we can see that a large number use the Dashboard button to login to the app, and those who haven’t tried it are primarily interested in pricing, followed by the Features page.
  • Primary use being to understand whether visitors are even scrolling down looking at your content or not.
    If on-site surveys indicate that visitors can’t find a piece of information when it exists on the page, run a scrollmap to see if people are scrolling down or not
  • The aim of great research should be to produce great hypotheses
    Great hypotheses are those that understand and take into account the core of your visitors’ behavior

×