Anúncio

Testing Roadmaps and Experiment Design

Web Optimisation Consultant at SiteSpect, Inc and Consultant at trsdigital Ltd em SiteSpect, Inc.
2 de Dec de 2015
Anúncio

Mais conteúdo relacionado

Apresentações para você(20)

Similar a Testing Roadmaps and Experiment Design(20)

Anúncio

Testing Roadmaps and Experiment Design

  1. TEST ROADMAPS How to win sales and influence success
  2. Hi I am Tim Stewart from trsdigital You might know starring roles such as: Commercial Manager for the largest English language motorcycle site in the world Consultant with the highest success rate, longest emails AND undefeated go-kart champion during Maxymiser’s highest growth years Web Optimization Consultant for SiteSpect with client’s such as well, er Connor you just met, Confused.com, Hotels.com and Guess Jeans EU Some may simply know me as that bloke who talked for 5 hours about Excel on the #MeasureFest training course yesterday The internet mostly knows me as @pjeedai on most social networks The p stands for Padawan as I am always learning INTRODUCTION @pjeedai
  3. Effective testing starts with the boarding process Vendor or Agency bringing on a client, an Internal team rolling out a new initiative or flying solo Its not as simple as adding the tool and taking off You can’t just go up the ramp, turn left to First Class and wait for the complementary peanuts BOARDING
  4. The energy when people talk about CRO can be breathtaking It may feel like Boarding is simply a question of hanging on and letting the wave push you There’s nothing more to do except enjoy the ride. HIGH ENERGY BOARDING
  5. But some stakeholders don’t have as much energy as others You need to understand the level of buy-in and competence of all the stakeholders It might mean your plans to roll out your testing program have to be adapted so they don’t overwhelm some parties. LOWER ENERGY BOARDING
  6. And some are just along for the ride MINIMAL ENGAGEMENT BOARDING
  7. BOARDING For initial boarding you should establish: • Who needs to be involved in the process • What responsibilities each person has • How you will roll out your testing plan Otherwise your life-changing trip might end with a bang … and even worse a Celine Dion song
  8. DISCOVERY Arguably the most important part of any roadmap process You find out what you are working with And what you need to work around.
  9. DISCOVERY: UNDERSTAND YOUR LAUNCH CONDITION Try understand the inertia you have to overcome What is currently weighing your business down? • How much weight do you need to shift? What is the shape of the business? • Where is money being made? • Which objectives have highest importance • What outcomes have the most volumes? • Which generate the most revenue? Can you quantify this? • How do web metrics report these values? • How closely do web metrics relate to outcomes?
  10. DISCOVERY: NOT JUST NUMBERS, ALSO RESOURCE Before looking at any website issues Be clear on the following: • What the business needs for take-off • What needs to be handled to achieve this Who looks after this? Who needs to be involved? What is required in terms of: • Development resources • Design resources • Sign-offs
  11. DISCOVERY: TRAJECTORY IS IMPORTANT Testing is often pitched as a way to • Bypass IT • Fast track past release schedules. It can be used for this. But care is needed. Marketers don’t want to hear it, but: • Checks • Balances • Sign-Off • Pre-flight checks are important! Too much push Too little direction Will mean a short, sharp trip A spectacular end to your plans
  12. Binary choices: • Leave a page or move forward • Add or Not Add • Checkout Now or Continue Shopping • Click to Read More • Sign up for our mailing list Where do the best opportunities lie? • What makes up the total? • Where are the biggest holes? • Where is the most money spent on traffic? Where is the signal loudest? • Like - Will continue • Don’t Like - Will go no further? Are there channel or device differences? DATA
  13. Data about how those choices happen • Click patterns • Heat maps • User Journey • Detailed step conversion tracking Data about users and their motivation • Never will Buys • Might Buys • Could be Persuadeds • Plan to Buy • Will Buy no matter what Do you have customer feedback? • User testing • Customer survey • Live Chat sessions • Personas SERIOUS DATA
  14. Quantitative and qualitative • Cohort and behaviour profiles • NPS • Customer Buying Patterns • Typical Basket composition • CRM and userID • BI: Margin per product, AOV, LTV, RPU Can you stitch it together? • Get an idea of themes • Areas to explore • Key templates • Decision points? • Customer Persona • Non-Customer Persona It is a term that is often misused Its not Big Data in terms of machine learning REALLY SERIOUS DATA
  15. But it is a hopefully large collection of data sets and useful intelligence And importantly it is not a fixed set of Data It will be added to over time • Through tests results • Through additional research • Through improved analytics Once your testing shows more detail around what drivers user choices you SHOULD be tweaking your measurement on site to get a clearer picture And obviously this new data should be fed back into the process and used to create and refine testing ideas and roadmap areas. NOT REALLY “BIG DATA” (BUT IT MIGHT BE LOTS) (This is actually Lore)
  16. It can be overwhelming But this is why Boarding and Discovery play such an important role. You aren’t on your own in this. You brought in the Business Intelligence team, the Department stakeholders, the Board, the HiPPOs When faced with a mountain of Data to look at it’s generally best to look for what makes the biggest difference to the company And look at the customer and analytics data to see what makes the biggest difference for the users. LOTS OF DATA CAN OVERWHELMING
  17. So if your stakeholders and customers have given you some clues… Start there. Be methodical. Drill down and work outwards to eventually the periphery of “other interesting stuff” when you come to focus on an area. Initially at least you are looking for a few places to start and some holes which need investigating INVESTIGATE YOUR DATA TO FIND SOME CLUES
  18. At this point you haven’t looked in any depth at any specific pages or processes You’ve looked at the business, the people in the business, the audience you can work with. IDEAS So you’ve established
  19. Who you need involved from the business What level they need to be involved The best way to communicate the strategy and the results IDEAS: WHO IS BEING INCLUDED? Stakeholders
  20. What they need for the business What they consider to be the critical results to meet the business strategy Do they have a timeline in mind? Is it reasonable? Fundamentally what you need to achieve from testing to ensure its continued support and growth. HiPPOs are not stupid. They are looking out for the best for the business. If you can show value and align your testing program with what the business needs to grow and succeed then HiPPOs will back you Which is how they became the Highest Paid People. And delivering a testing roadmap that delivers success for the business will make you more Highly Paid too IDEAS: THE HIPPO’S IDEAS DO MATTER The HiPPO
  21. The Audience is your sample. It is what you have to test against It is their behaviour that you want to influence. • How much you can test? • How it can be segmented? What is the composition? New, Returning, Regular Purchasers, Irregular, Distress Purchase, Subscription. They all have different motivations and will react differently What is the shape? Frequency, Recency, Device preference Are users doing the same thing on every visit? Do different devices get used for different purposes? Are there obvious places where mobile outperforms or underperforms Desktop? WHY? IDEAS: WHO AND HOW MANY TO TEST AGAINST Audience
  22. How many of these have audience? Sometimes area of great internal debate have much lower engagement or user volumes to test against. Sometimes you want to measure increase to Final Sale But there simply aren’t enough volumes Or the Final Sale is 4 visits, 2 devices and 18 pages after the area you want to test. So is Final Sale the most closely related metric to what you are changing? You will, eventually, influence it. You may want to track it every test for a benchmark But unless you are testing the page directly before and have large volumes of commonly distributed purchases every day.. It will limit your testing ability It should not be your sole focus or your primary decision metric on every test. If you focus on just the cash register but don’t stock the shelves, promote the offers, price match competitors, see how users navigate the aisles… Your Cash Register is going to remain too quiet and will provide you very little intelligence on why. IDEAS: WHERE AND WHAT NEEDS FOCUS Places
  23. The potential for Win in those places and with the Audience identified means you can start to think on priorities You also start to understand the size of the challenge you have to convince the Audience to change their behaviour You will NOT go from 0.3% conversion rate to 3% easily An idea of the value? It won’t be perfect but you need an idea of the value, in revenue, happy customers, business objectives (acquisition, AoV, RPU etc) Possibly from the data alone you will already have a clear idea on some of the questions you need to answer and options you can test Which of these are traffic wins? (more people into a key area) Which of these are business wins? (more sales, greater value, the “right” product being seen/chosen) Do these align or are they always separate tracks? IDEAS: WHICH ARE BIG ENOUGH TO MATTER Scale of Opportunity & Challenges
  24. What are the targets you have? What budgets? What current actuals? Want != Need I want more sales now, for less spend Translates to I need to find a way to increase conversion from the same audience I need to identify where this is being impacted and reduce this This NEEDS time, experimentation and commitment. If one test gave all the answers, halved the CPA .. Then there wouldn’t be a need for a Roadmap. IDEAS: WHAT TARGETS NEED TO BE MET What the Business wants Hopefully what the business actually needs (these two rarely align)
  25. Look at the pages with fresh eyes. Think – What do WE want here? Think – What does the CUSTOMER want here? Look – What does the Customer ACTUALLY do? Why is this different to what we want/intend? (it will be, customers are usually a LOT less keen to give you all their money immediately) IDEAS: THINK LIKE A CUSTOMER What the Customer DOESN’T want
  26. Go to the places your data has shown need attention Go to the places the business and stakeholders want to improve TRY to break the user journey. Screen grab every error, every misstep, every button/link/option that is NOT move forward Look for friction, miscommunication, ways where Where Next is not clearly answered Blur your eyes, see if you can still see a CTA Try different screen sizes, is that important information above fold (spoiler: it still matters, even if you can encourage people to scroll) The UX book is called Don’t Make Me Think. Do you? Do you make the customer have to *shudder* think? And Read? Can they do it in < 3 seconds from landing? Then think about how you can change that, some alternative ways to handle those choice points IDEAS: GET CREATIVE
  27. And you will end up with ideas, lots and lots of ideas. It’s probably a good idea to write them down somewhere. LOTS AND LOTS AND LOTS OF IDEAS
  28. You’ve sat down • Got the business objective(s) • Got the web stats which relate to that objective • Got a budget • Got a timeline • Got lots and lots and lots of ideas Just a question of pointing it in the right direction and heading to Success-ville? A ROADMAP TO SUCCESS
  29. If that is the conclusion of your boarding and discovery – Get Ideas, Test Ideas, Epic Win! I have news for you. You Shipment of Fail has arrived. Because you may have created the biggest shopping list of issues ever and “testing will fix that” But its at this point a lot of businesses choke. And I bet at this stage some of your boarders have run out of energy and you won’t have the traffic or resource to test everything. EXPECTATION VS. DELIVERY
  30. Too many, dare I say it, Ideas Because we can’t test Ideas. Ideas are formed into hypotheses. Hypotheses are formed into tests. Because the road is never that smooth. The choke points of resource and traffic are always too small. If your boarding and discovery showed that you have everything you need, no impedance, no need for alternatives or expectation that you will deviate You are going to have a bad time EXPECTATION VS. DELIVERY
  31. Set a destination that looks more achievable for your circumstances Plan a more realistic way to get there with the resources you have PROCESS TO RELIABLY REACH SUCCESSA ROADMAP TO SUCCESS?
  32. PROCESS TO RELIABLY REACH SUCCESSA ROADMAP TO SUCCESS? Now we all work in digital. This isn’t anything new, or at least it shouldn’t be. The Board & hungry HiPPOs may think this is what they will get – a direct line to Success The reality is that you won’t fly first class to success even with the best Boarding & Discovery process
  33. PROCESS TO RELIABLY REACH SUCCESSA ROADMAP TO SUCCESS? The reality of release schedules, sign off delays, legal & brand policy departments getting involved will be specific to your situation and will always be evolving. Your tests won’t always work out the way you expect; you might need to test again and check results, stop testing in one area to work in another, account for changes in market or tactical need etc. It is very likely there will be congestion on the ideal route, diversions, unplanned stop-overs & lost luggage If you only have a single focus plan, only test when conditions are perfect then you will step your way there. But slowly, through adverse conditions. You will back step regularly, roadblocks can & will stop you completely unless you have something prepared.
  34. PROCESS TO RELIABLY REACH SUCCESS Now obviously you don’t know, what you don’t know. You can’t draw a plan to account for every possible outcome upfront, no matter what Data you have But you can and should have a few alternative paths to explore that will also get you to your destination A successful Roadmap will combine multiple areas of investigation, several paths forward to your objectives The idea is to have a Process that keeps you testing somewhere & keeps momentum in your Roadmap
  35. What we need are tests. What a test needs is a reason to exist; a Hypothesis You need to have a think, cogitate on your next move. Think a few moves ahead COGITATE: PLAN YOUR MOVES So I said earlier that we don’t really want Ideas. But initially we do need them, we just need to craft them into hypotheses We need to put them together. Chew them over. Have a think. Make a list
  36. IDEAS WILL COME IN MANY FORMS, SOME EASIER TO COMPILE THAN OTHERS Ideas will come in from all stakeholders and Persons with Opinions (Highly Paid or not) They will come in many different formats and need to be consolidated into a big pile of “stuff we could look at” And you will welcome these (because understanding who and what was part of the process and involvement helps you get traction) COLLECT & COLLATE An unordered list of business problems and design wishlists A set of ideas, grouped into tracks Some simple time & sequence planning A fully quantified spreadsheet, with tracks matching business areas and priorities accounting for release schedules An email with only basic ideas
  37. These ideas need to be refined into something you can actually test with: Hypotheses Structured Test Concepts not just “make it better and more punchy” What parts do you have? What parts are missing? CONSOLIDATE: ORGANISE THE PUZZLE PIECES These ideas are likely to be pieces of the puzzle, parts of a hypothesis, not a complete test The next phase therefore is to pull these together into a pile of parts you can work with
  38. You won’t get this right on the first pass. Or the second. Like every part of testing this is an ongoing optimisation. Permanently re-adjusting and tweaking until you have something that works But initially at least… good enough will do, even if there are some bits that just don’t look quite right. Split Ideas into groups; themes, site sections, areas of business interest Divide sometimes (often?) conflated and unviable ideas into discrete achievable tests Match available hours, ability and traffic to achieve the most appropriate mix CONSOLIDATE: SEE WHAT FITS
  39. Just because you can, doesn’t mean you should EXPERIMENTAL DESIGN TIME FOR A QUICK DETOUR INTO EXPERIMENT DESIGN Seeing as we are talking about turning ideas into tests. So a quick refresher on Experimental design
  40. THE PROBLEM WITH MOST EXPERIMENT DESIGNS When given a tool that allows you to change almost anything And a list of business problems which, if fixed, could make LOTS more money The typical reaction is to dive in and try to do it all in one big test… The typical result is #epicfail
  41. Hypothesis Kit 3 Craig Sullivan @OptimiseOrDie Colin Mcfarland, Lukas Vermeer, Rik Higham, Doug Hall, Michael Aagard and many others who helped shape this https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc HYPOTHESES Simple Kit 1. Because we saw (data/feedback) 2. We expect that (change) will cause (impact) 3. We’ll measure this using (data metric) Advanced Kit 1. Because we saw (qual & quant data) 2. We expect that (change) for (population) will cause (impact(s)) 3. We expect to see (data metric(s) change) over a period of (x business cycles) “We don’t like this page” is NOT a valid Hypothesis There are several ways to frame this but here is one I like for it’s clarity and simplicity
  42. A lot of stakeholders don’t think in terms of hypotheses They’ve focused on the business problems and who actually has to do this So a Scope, Brief, Qualifying Criteria for a test may be better explained if you can answer the 5W1H questions QUALIFYING CRITERIA What • Do we want to achieve? Where • Are the changes to be made? Why • Based on what data? • Based on what need? Who • Is this aimed at? • Is responsible for sign off? When • Can we do this? • Is this running? How • Is that measured? How will that show in the metrics we have defined • Much difference could be made?
  43. DECISION METRICS: WHAT DOES SUCCESS LOOK LIKE? Decision Metrics • Appropriate to the question you are asking? • Directly relate to Test Objective? What are you changing? • Expected effect • What should users do more of, more often? Unexpected effect • What if they do less? • Or worse… No Significance? • What if it is substitution? Primary Call To Action or Outcome • Ancillary Outcomes • Value or Meta Metrics
  44. The most effective metric is the closest Next Action for a user who has seen your change • What are you changing? • Why? • What should the user do if it works? • What could a user do if it doesn’t? Measure that as your main signal Sometimes users do less of your main metric but substitute another If you can, measure that too Sometime the main metric increases but an alternate path to Next Action decreases This can mean no NET gain, what you have is substitution not incremental increase So measure that too. You want a clear result It doesn’t need to be Up, sometimes quantifying the Risk of Down is critical The only failed test is one where you don’t learn something UNCLEAR METRIC DEFINITIONS: UNCLEAR RESULTS
  45. • Weighting • Priority • Potential • Importance • Ease • Reusability • Dependencies • Restrictions TRIAGE Many methods to prioritise YOUR tests to suit YOUR situation, limitations and objectives You know what they are, right? After all we established this in Boarding and Discovery… didn’t we?
  46. Emotion and tactical need will enter into it But if your whole roadmap is led by whim and gut feel it is easy to go off course PRIORITISATION Many people will think Priority is decided by which HiPPO is the most loudy shouty Whilst this can be accurate, it is not the most reliable way to plan an effective test roadmap
  47. PRIORITISATION It is the HiPPOs job to focus on • The end result • Big difference for the whole company • Technically difficult site-wide changes They will want BIG RESULTS quickly They need to quantify risk or reward But if you wait for only those tests… Your roadmap will stall Test for intelligence on what wins You can’t test to only get wins Explain, carefully, sometimes slowly: • Some tests are small • Some are far away
  48. • Keep a central record of what is agreed • Keep a revision history of that over time Check it from time to time to remind yourself how you have had to re-plan Remind yourself and stakeholders that changing plans is expected and required PRIORITISATION This is where a framework and Discovery are very useful. Don’t get into emotional debate Quantify what happens when, why and what is needed to make later tests happen sooner.
  49. CRO IS UNDERSOLD. IT CAN DO MORE THAN “JUST” GET WINS But the ease with which consistent victories are achieved is massively oversold Technology will not solve all the issues What makes most difference is the people and the teamwork SUPERHEROES OF CRO We like to portray CRO as heroes of the business, fixing bad UX and saving the world But, a confession. In reality it is not the company-saving Alpha-male magic beans we were sold
  50. DIFFERENT BREED OF HERO In fact it needs a different kind of hero. One that is much more … office-based & organised And wears a lot less lycra
  51. Working through what is needed, not just what looks good requires a methodical approach You can run around smashing heads and breaking hearts but it is easy to make more noise than value The biggest gains are made from long-term cultural shift in behaviour It doesn’t mean you have to go slow. With a good repeatable process you can go faster Because you already have a list of targets that add up to a clear goal You waste less time than looking around for only the headline grabbing big battles after you recover from the last DIFFERENT BREED OF HERO That’s not to say that this different breed of heroes can’t kick ass With the right approach and the right tools they are very much in the world-saving business They just do so with a little more of a measured approach and less collateral damage.
  52. PROJECT MANAGEMENT In fact we aren’t talking about a revolutionary approach When it comes to Roadmapping for success an essential tool is… Project Management I recommend and use www.liquidplanner.com but there are many options out there
  53. Excel, Smartsheet, Whiteboard, Piece of Paper. Ultimately its what fits and what you can use. But something you can share with the team and easily update is recommended PROJECT MANAGEMENT A good basic (and cheap) option is www.smartsheet.com which offers several templates ProTip: You can also link sheets so the Projects in the Gantt links to items from your Priority/Ideas sheet It also has good Google integration and allows creation of Forms for adding new items from Stakeholders
  54. Task lists don’t give you an oversight of how all tests fit together Basic lists of Projects won’t show you what Task is needed next from each person Simple Gantts or Card Views don’t let you move Tasks independently of Projects It is entirely likely (and recommended) to be in planning phase of Test 4 and design phase of Test 3 when Test 1 is live and Test 2 is in Final QA PROJECT MANAGEMENT Task based tools like Asana or Basecamp can also work, as can card style tools like Trello. These type of tools are simple to use but lack oversight on scheduling resource over multiple tests (The image is actually Card view from Liquid Planner – simple view, underpinned by a proper scheduling system)
  55. PROJECT MANAGEMENT Liquid Planner works well for CRO because it is flexible & any plan needs to be flexible • Several ways to look at, reprioritise and reschedule multiple overlapping projects • Move tasks up the priority list & work on them out of sequence when you need to work ahead • Every time you update priority it re-calculates the likely delivery for each task & project
  56. COMMON SENSE: SO RARE IT’S A SUPERPOWER Prioritise and plan for delivery of Tests, not just ideas. How MUCH you do depends on your setup & situation: Resource Capacity • Traffic and sample size for metrics • Developer availability • Analyst & Planning availability • Design availability Resource Capability • Technical ability within the team • Technical function of the Testing tool • Experiment planning ability • Analytics and Statistical ability Want to go fast and run small tests. Plan for that Want to plan more and test for impact? Do that Want BIG TESTS and quickly. Be big. Pay big (still fail lots) Ultimately use Common Sense. It’s still so rare it’s a super power
  57. Multiple Paths Varying Timelines Changing Complexity Plans will change Embrace that. Simply planning Tests will breed new test ideas Results will create more need to explore in some areas than originally expected. Results will kill whole planned sets of tests if they are shown to have less value than anticipated. EVOLVE Test 1 Mobile 1 Mobile 1a Mobile 2 Mobile 3 Mobile LP Landing Pages LP 1 LP 2 LP 3 LP 4 LP 2a LP 2b Checkout Checkout 1 Checkout 1a Checkout 1b Checkout 2 Checkout 3 Checkout 4 Checkout 5 Test 1a
  58. Nothing is set in Stone • Revise those plans and documents after each update • Document each revision, track what takes time • The aim is to improve the process as well as the site It takes an investment – a commitment on time You can’t just cut out fat to make efficiencies and optimize You have to measure and invest where your system is lacking Anyone says otherwise is telling Porkies EVOLVE
  59. For every test on the Working List Create Test Documentation • Document Your Hypotheses • Confirm decision metrics • Record what you intended to happen • Scope how you want that to be built • Specify URLs to include/exclude • Specify Segments to include/exclude If you came back to this in a year could you explain to a new starter what was intended? Memory is fallible. If you had to come back to your developer in a week and explain why it failed QA.. Is it in here? If you’re “just trying something” without data, objectives, a Hypothesis and Metrics…It’s not a test. You can build it and run it. But it’s not a test EXECUTION: DOCUMENT EVERY TEST Buy Today Buy Today Buy Today click Why is it bigger? How much bigger? What should users do more of? Is this on all pages? All devices? Image or CSS? What does this do to nearby elements?
  60. Wireframes • Blueprint your changes • Show what it should look like Take a closer look at the page • Look at what else could happen • Is the change clear? • Could something else be clicked on? • Are you measuring that too? This is your blueprint, the concept to get approved before you spend time and money on design. This is the idea your stakeholders must buy into, your designer must understand, your developer must build. Have you shown all the states the page can be in for the test? Does it work in all cases? EXECUTION: BLUEPRINT THE CHANGES clickclick click clickclick click clickclick click
  61. Issue Resolution • Plan for Use cases • Try to break the page • Try it logged in, not logged in • Checkout with nothing in basket • Submit forms with fields empty Build and try to find omissions • QA • QA again Hold a pre-mortem Blame no-one, open mind and think: • If in n weeks you are reporting a failure • Why did it fail? What broke? What is the cause of death you have to pronounce? • Could it have been avoided? Produce a conclusion report for EVERY test. What worked, what didn’t EXECUTION: PRE AND POST MORTEM EVERY TEST
  62. Test Results should change your plans • Regular reviews • Is this still in line with strategy? • Could we go faster? Should we go slower? • Test Results • Did the result show further potential? • Did the result show greater risk? • Test non-results • Why is it not clear? • Did the user not care enough? • Did one segment “outvote” another? Don’t underestimate Tactical Issues • Releases • Team Changes • Market conditions • Seasonality ADAPT Review Results Non- Results Releases Team Changes Tactical issues
  63. CONCLUSION Bet you thought this would never end, nearly time for the coffee break We’ll finish with a quick recap
  64. BUILD FROM GOOD BOARDING Good Boarding process No Boarding process GOOD BOARDING = UNDERSTAND THE PEOPLE
  65. DISCOVERY PHASE ENSURES SUCCESSFUL TAKE-OFF DISCOVERY = UNDERSTAND THE ENVIRONMENT AND EQUIPMENT
  66. BE SERIOUS ABOUT YOUR DATA DATA DRIVEN DECISIONS, OTHERWISE IT’S GUESSWORK
  67. All too easy to think data and a plan has to be perfect to start All you need is Good Enough The process is to refine all of this as you go along TOO MUCH CAN BE INTOXICATING DECIDE – DON’T JUST COLLECT DATA AND TAKE NO ACTION
  68. BUT GET IT RIGHT, AND YOU’RE LAUGHING STRIKE A PRAGMATIC BALANCE THAT SUITS YOUR SITUATION
  69. IDEAS FROM BUSINESS, USERS AND DATA BE CREATIVE TRY TO BREAK YOUR USER JOURNEYS IT IS NEVER THE FLOW YOU INTENDED
  70. BALANCE YOUR PRIORITIES BALANCE TO SUIT YOUR CAPACITY
  71. QUALIFY WHAT MAKES AN IDEA A TEST CANDIDATE MAKE EVERY IDEA EARN ITS PLACE ON THE LIST
  72. QUANTIFY YOUR PRIORITIES TO THE HIPPOS Calmly explain; facts and data back up strong arguments Anger, fear, aggression lead to the dark side (turning off the lights has a similar effect) OPEN COMMUNICATION WITH STAKEHOLDERS USE DATA TO BACK UP YOUR POSITION
  73. EXPLAIN HIGH PRIORITY DOESN’T MEAN “NEXT TEST” BY DEFINITION EVERYTHING IS ASAP Because Soon and Possible are subjective variables. Use Common Sense
  74. BE A NEW BREED OF CREDIBLE HERO DON’T GET CAUGHT UP IN YOUR OWN LEGEND Be “Get things done” rather than “Look at me”
  75. PROJECT MANAGEMENT IS NOT A DIRTY WORD A TESTING PLAN REQUIRES PLANNING Don’t forget Common Sense Common Sense and pragmatism go a long way
  76. RECORD ALL PLANS (BUT NOTHING IS FROZEN) WRITE IT DOWN, BUT BE PREPARED AND HAPPY TO CHANGE IT
  77. ADAPT: LEARN FROM BAD RESULTS BAD RESULTS ARE STILL RESULTS, LEARN FROM THEM
  78. EVOLVE: TEST FOR INTELLIGENCE NOT JUST RESULTS BE AGILE: SEE AN UNEXPECTED OPPORTUNITY THEN EXPLOIT NEW AREAS
  79. EFFECTIVE ROADMAPS @pjeedai www.trsdigital.com

Notas do Editor

  1. 1. MVT AND AB TESTING: ROADMAP FOR SUCCESS
  2. 2. INTRODUCTION Hi I am Tim Stewart from trsdigital You might know starring roles such as: Commercial Manager for the largest English language motorcycle site in the world Consultant with the highest success rate, longest emails AND undefeated go-kart champion during Maxymiser’s highest growth years Web Optimization Consultant for SiteSpect with client’s such as well, er Connor you just met, Confused.com, Hotels.com and Guess Jeans EU Some may simply know me as that bloke who talked for 5 hours about Excel on the #MeasureFest training course yesterday The internet mostly knows me as @pjeedai on most social networks The p stands for Padawan as I am always learning
  3. 3. BOARDING Effective testing starts with the boarding process Vendor or Agency bringing on a client, an Internal team rolling out a new initiative or flying solo Its not as simple as adding the tool and taking off You can’t just go up the ramp, turn left to First Class and wait for the complementary peanuts
  4. 4. HIGH ENERGY BOARDING The energy when people talk about CRO can be breathtaking It may feel like Boarding is simply a question of hanging on and letting the wave push you There’s nothing more to do except enjoy the ride.
  5. 5. LOW ENERGY BOARDING But some stakeholders don’t have as much energy as others You need to understand the level of buy-in and competence of all the stakeholders It might mean your plans to roll out your testing program have to be adapted so they don’t overwhelm some parties.
  6. 6. MINIMAL ENGAGEMENT BOARDING And some are just along for the ride
  7. 7. AGREE A PROCESS OR FAIL BEFORE YOU START Having an agreed process for Boarding is recommended – even a simple one Identifying responsibilities and requirements is important
  8. 8. DISCOVERY PHASE Arguably the most important part of any roadmap process You find out what you are working with And what you need to work around.
  9. 9. DISCOVERY IS ABOUT THE BUSINESS
  10. 10. THE BUSINESS IS NOT JUST THE NUMBERS BUT THE RESOURCE What is the holistic environment you need to consider? It will vary for every company, there is no single answer You need to know: The congestion Road conditions If the vehicle is suitable If you are fully fuelled for the journey All your passengers are on board Are you ready to launch?
  11. 11. POOR DISCOVERY LEADS TO DISASTER How fast your plan can take off depends very much on What you have to achieve that On what velocity you need to make a difference If you have enough push to overcome resistance Establish the people, the potential, the limitations This is the point where these things can be investigated and mitigated Making every following step easier to achieve.
  12. 12. DATA DRIVEN DECISIONS Don’t just start with the headline site conversion rate and pick a desired improvement Look at how that rate changes over a year, over a week, over channels, over devices. Your tests will be trying to move the needle enough to be measureable You need to understand what a typical variation in conversion looks like You need to be clear on what buying patterns and buying cycles to test against You can’t affect the whole site conversion in one test. You can…(it is possible, I’ve done it) but, really you can’t (rely on that) What you can do is shift the needle enough on one step of the buying process in one lucky test that it makes a big enough difference to the whole But those opportunities are rare. Usually those holes were big enough to be spotted and if not fixed then improved already. So you need Data. Analytics on what makes up those steps towards a sale or macro conversion You need to know how much you have to measure against. Understand what you think is the current benchmark How much signal? How much noise? How much money is being spent in areas that underperform? How much money is being made in areas that perform OK but you would like more? What numbers do you have to show where the worst friction happens for a user Is it the same for all users? Return, new, frequent flyers, researchers, serial purchasers, one-off interest? Do you see different choke points for users on different devices?
  13. 13. DATA ABOUT HOW THOSE NUMBERS COME TO HAPPEN What is the click pattern, heat map, user journey for Never will Buys, Might Buys, Could be Persuadeds, Plan to Buy, Will Buy no matter what Do you have analytics on micro events? Heatmaps, user testing, customer survey, Live Chat sessions, Personas? You might have none, some or all of this and more But move away from only money or goal focused KPIs Look at the behaviours that correlate with Buying and Nearly Buying and Not Buying Anything that will help understand how those holes in the conversion process are created You look at buyers to see what they like But also look at the users who didn’t buy, how is their behaviour different? Can you correct that without annoying the buyers?
  14. 14. QUANTITATIVE AND QUALITATIVE – HOW DOES QUALITY RELATE TO ACTIONS? Can you stitch it together to get an idea of themes, areas to explore, key templates or decision points? Its not really Big Data in terms of machine learning. Its several sets of small to medium data But it’s a lot more data to look at than just Headline site conversion rate A typically optimistic forecast of 20% YoY, regardless of circumstance Because you don’t just want to look at pages you and the board target and assume a quick set of best practice tests will work You need to look at what causes the user to not take the actions you want. And you need to look at all the potential friction points that show the user is not reacting the way you intend
  15. 15. IT IS NOT BIG DATA But it is a hopefully large collection of data sets and useful intelligence And importantly it is not a fixed set of Data It will be added to over time Through tests results Through additional research Through improved analytics Once your testing shows more detail around what drivers user choices you SHOULD be tweaking your measurement on site to get a clearer picture And obviously this new data should be fed back into the process and used to create and refine testing ideas and roadmap areas.
  16. 16. LOTS OF DATA; WHERE DO YOU EVEN START? It can be overwhelming But this is why Boarding and Discovery play such an important role. You aren’t on your own in this. You brought in the Business Intelligence team, the Department stakeholders, the Board, the HiPPOs When faced with a mountain of Data to look at it’s generally best to look for what makes the biggest difference to the company And look at the customer and analytics data to see what makes the biggest difference for the users.
  17. 17. START WITH CLUES FROM THE BUSINESS AND LOUDEST SIGNALS FROM THE USERS So if your stakeholders and customers have given you some clues… Start there. Be methodical. Drill down and work outwards to eventually the periphery of “other interesting stuff” when you come to focus on an area. Initially at least you are looking for a few places to start and some holes which need investigating
  18. 18. TIME TO CHECK SEE WHAT YOU HAVE At this point you haven’t looked in any depth at any specific pages or processes You’ve looked at the business, the people in the business, the audience you can work with.
  19. 19. STAKEHOLDERS Who you need involved from the business What level they need to be involved The best way to communicate the strategy and the results
  20. 20. THE HIPPO What they need for the business What they consider to be the critical results to meet the business strategy Do they have a timeline in mind? Is it reasonable? Fundamentally what you need to achieve from testing to ensure its continued support and growth. HiPPOs are not stupid. They are looking out for the best for the business. If you can show value and align your testing program with what the business needs to grow and succeed then HiPPOs will back you Which is how they became the Highest Paid People. And delivering a testing roadmap that delivers success for the business will make you more Highly Paid too
  21. 21. AUDIENCE The Audience is your sample. It is what you have to test against It is their behaviour that you want to influence. How much you can test? How it can be segmented? What is the composition? New, Returning, Regular Purchasers, Irregular, Distress Purchase, Subscription. They all have different motivations and will react differently What is the shape? Frequency, Recency, Device preference Are users doing the same thing on every visit? Do different devices get used for different purposes? Are there obvious places where mobile outperforms or underperforms Desktop? WHY?
  22. 22. PROBLEM PAGES, PROBLEM PROCESSES How many of these have audience? Sometimes area of great internal debate have much lower engagement or user volumes to test against. Sometimes you want to measure increase to Final Sale But there simply aren’t enough volumes Or the Final Sale is 4 visits, 2 devices and 18 pages after the area you want to test. So is Final Sale the most closely related metric to what you are changing? You will, eventually, influence it. You may want to track it every test for a benchmark But unless you are testing the page directly before and have large volumes of commonly distributed purchases every day.. It will limit your testing ability It should not be your sole focus or your primary decision metric on every test. If you focus on just the cash register but don’t stock the shelves, promote the offers, price match competitors, see how users navigate the aisles… Your Cash Register is going to remain too quiet and will provide you very little intelligence on why.
  23. 23. THE SIZE OF THE HOLES, THE VALUE OF THE POTENTIAL WINS. The potential for Win in those places and with the Audience identified means you can start to think on priorities You also start to understand the size of the challenge you have to convince the Audience to change their behaviour You will NOT go from 0.3% conversion rate to 3% easily An idea of the value? It won’t be perfect but you need an idea of the value, in revenue, happy customers, business objectives (acquisition, AoV, RPU etc) Possibly from the data alone you will already have a clear idea on some of the questions you need to answer and options you can test Which of these are traffic wins? (more people into a key area) Which of these are business wins? (more sales, greater value, the “right” product being seen/chosen) Do these align or are they always separate tracks?
  24. 24. WHAT THE BUSINESS WANTS, WHAT THE BUSINESS NEEDS What are the targets you have? What budgets? What current actuals? Want != Need I want more sales now, for less spend Translates to I need to find a way to increase conversion from the same audience I need to identify where this is being impacted and reduce this This NEEDS time, experimentation and commitment. If one test gave all the answers, halved the CPA .. Then there wouldn’t be a need for a Roadmap.
  25. 25. UNDERSTAND WHAT USERS DON’T WANT Look at the pages with fresh eyes. Think – What do WE want here? Think – What does the CUSTOMER want here? Look – What does the Customer ACTUALLY do? Why is this different to what we want/intend? (it will be, customers are usually a LOT less keen to give you all their money immediately)
  26. 26. BE CREATIVE: LOOK AT THE SITE, GET IDEAS Go to the places your data has shown need attention Go to the places the business and stakeholders want to improve TRY to break the user journey. Screen grab every error, every misstep, every button/link/option that is NOT move forward Look for friction, miscommunication, ways where Where Next is not clearly answered Blur your eyes, see if you can still see a CTA Try different screen sizes, is that important information above fold (spoiler: it still matters, even if you can encourage people to scroll) The UX book is called Don’t Make Me Think. Do you? Do you make the customer have to *shudder* think? And Read? Can they do it in < 3 seconds from landing? Then think about how you can change that, some alternative ways to handle those choice points
  27. 27. THERE WILL BE LOTS OF IDEAS And you will end up with ideas, lots and lots of ideas. It’s probably a good idea to write them down somewhere.
  28. 28. SO THAT IS ALL YOU NEED, RIGHT? You’ve sat down Got the business objective(s) Got the web stats which relate to that objective Got a budget Got a timeline Got lots and lots and lots of ideas Just a question of pointing it in the right direction and heading to Success-ville?
  29. 29. BAD NEWS – EXPECTATIONS MAY NOT MATCH REALITY If that is the conclusion of your boarding and discovery – Get Ideas, Test Ideas, Epic Win! I have news for you. You Shipment of Fail has arrived. Because you may have created the biggest shopping list of issues ever and “testing will fix that” But its at this point a lot of businesses choke. And I bet at this stage some of your boarders have run out of energy and you won’t have the traffic or resource to test everything.
  30. 30. TOO MUCH DATA, NOT ENOUGH FOCUS ON DELIVERY Too many, dare I say it, Ideas Because we can’t test Ideas. Ideas are formed into hypotheses. Hypotheses are formed into tests. Because the road is never that smooth. The choke points of resource and traffic are always too small. If your boarding and discovery showed that you have everything you need, no impedance, no need for alternatives or expectation that you will deviate You are going to have a bad time
  31. 31. SUCCESS REMAINS THE GOAL BUT CONSIDER A MORE REALISTIC APPROACH Set a destination that looks more achievable for your circumstances Plan a more realistic way to get there with the resources you have
  32. 32. A STRAIGHT LINE IS NOT A ROADMAP It summarises your objective, it shows direction of your strategy But it does not account for the tactics needed on the way to adjust for the conditions
  33. 33. REALITY CHECK A test plan without planning is a very inefficient way to get to your destination
  34. 34. DEVELOP A PROCESS FOR DEALING WITH ADVERSITY Plan alternative approaches Consider choke points and what tactics could be used
  35. 35. A BIG PILE OF IDEAS WILL CHOKE YOUR ROADMAP What we need are tests. What a test needs is a reason to exist; a Hypothesis You need to have a think, cogitate on your next move. Think a few moves ahead
  36. 36. IDEAS WILL COME IN MANY FORMS, SOME EASIER TO COMPILE THAN OTHERS Ideas will come in from all stakeholders and Persons with Opinions (Highly Paid or not) They will come in many different formats and need to be consolidated into a big pile of “stuff we could look at” And you will welcome these (because understanding who and what was part of the process and involvement helps you get traction)
  37. 37. CHECK OVER WHAT YOU HAVE COLLATED These ideas need to be refined into something you can actually test with: Hypotheses Structured Test Concepts not just “make it better and more punchy” What parts do you have? What parts are missing?
  38. 38. COMBINE THE PIECES: THIS IS ALSO AN ITERATIVE PROCESS You won’t get this right on the first pass. Or the second. Like every part of testing this is an ongoing optimisation. Permanently re-adjusting and tweaking until you have something that works But initially at least… good enough will do, even if there are some bits that just don’t look quite right.
  39. 39. TIME FOR A QUICK DETOUR INTO EXPERIMENT DESIGN Seeing as we are talking about turning ideas into tests. So a quick refresher on Experimental design
  40. 40. YOU CAN CHANGE ALL THE THINGS. YOU SHOULDN’T The problem with changing all the things is there is no clear hypothesis No Focus for the test. Therefore no Answer that you can get Or, if you do sit down and work out a hypothesis and counter hypothesis for every problem on the site then the test is too large to run or analyse.
  41. 41. USE A HYPOTHESIS FRAMEWORK A couple of great ways to frame what an idea needs to have before it can become the basis for a test.
  42. 42. DEFINE CRITERIA FOR WHAT MAKES A “TEST” IN YOUR BUSINESS A lot of stakeholders don’t think in terms of hypotheses They’ve focused on the business problems and who actually has to do this So a Scope, Brief, Qualifying Criteria for a test may be better explained if you can answer the 5W1H questions
  43. 43. MEASURE THE RIGHT THING The most effective metric is the closest Next Action for a user who has seen your change What are you changing? Why? What should the user do if it works? What could a user do if it doesn’t? Measure that as your main signal Sometimes users do less of your main metric but substitute another If you can, measure that too Sometime the main metric increases but an alternate path to Next Action decreases This can mean no NET gain, what you have is substitution not incremental increase So measure that too.
  44. 44. POOR TEST STRUCTURE & POOR METRIC SELECTION = UNCLEAR RESULTS You want a clear result It doesn’t need to be Up, sometimes quantifying the Risk of Down is critical The only failed test is one where you don’t learn something
  45. 45. SORT IT ALL OUT http://www.processexcellencenetwork.com/lean-six-sigma-business-transformation/articles/process-excellence-methodologies-using-prioritizat/ http://conversionxl.com/3-frameworks-help-prioritize-conduct-conversion-testing/ PIE or PIER is a simple option; also see Six Sigma, FUWDM, Plan/Measure/Improve frameworks or roll your own. Pick one that works for you, adapt it as needed. Explain to Stakeholders how priority will be judged Adjust the weight of each factor as needed, update regularly Or be Agile, have a regular stand-up meeting and argue a lot debate Sprints & minimum deliverables However you do it you want a Working List of 3 to 10 tests which are being planned, scoped, mocked up, developed, QA’ed, QA’ed again and Analysed If you are working in a team you don’t need to wait for a test to complete, go to another area and start planning the next one whilst the first two are in Build and QA If you are on your own, rather than pointlessly checking results of a live test 5 times a day (we all do) get on with planning a new test and QA the next one.
  46. 46. PRIORITISATION SHOULD BE METHODICAL Emotion and tactical need will enter into it But if your whole roadmap is led by whim and gut feel it is easy to go off course
  47. 47. HAVE A MIX OF SIZE AND COMPLEXITY We talked about a number of threads of enquiry, alternative routes. You need to plan a mix of all types of test Tests you can do now Tests you can plan ahead for Tests which will only be added later IF preceding results show those later tests are worth investigating. Keep refreshing that list Killed something as it turned out to be lower priority or too hard right now? Bring forward 2 or 3 potential candidates and try them on for fit in the hole you made in the roadmap
  48. 48. QUANTIFY AND RECORD YOUR CHOICES Keep a central record of what is agreed Keep a revision history of that over time Check it from time to time to remind yourself how you have had to re-plan Remind yourself and stakeholders that changing plans is expected and required
  49. 49. CRO IS UNDERSOLD. IT CAN DO MORE THAN “JUST” GET WINS But the ease with which consistent victories are achieved is massively oversold Technology will not solve all the issues What makes most difference is the people and the teamwork
  50. 50. PLANNING, ORGANISATION, DOCUMENTATION. IS THAT SUPERHERO STUFF? This is not a world of glamour and excitement In fact it sounds a lot like …. Project Management And … ick… Common Sense
  51. 51. DON’T UNDERESTIMATE THE QUIET MAN Working through what is needed, not just what looks good requires a methodical approach You can run around smashing heads and breaking hearts but it is easy to make more noise than value The biggest gains are made from long-term cultural shift in behaviour It doesn’t mean you have to go slow. With a good repeatable process you can go faster Because you already have a list of targets that add up to a clear goal You waste less time than looking around for only the headline grabbing big battles after you recover from the last
  52. 52. PLAN TO TURN IDEAS AND OBJECTIVES INTO PROJECTS AND DELIVERY So we aren’t talking about the glamourous super hero company saving Alpha male activity we were sold This sounds a lot like …. Project Management And … ick… Common Sense
  53. 53. EVEN A SIMPLE GANTT WILL HELP Excel, Smartsheet, Whiteboard, Piece of Paper. Ultimately its what fits and what you can use. But something you can share with the team and easily update is recommended
  54. 54. PRIORITY NEEDS TO BE FOR PROJECTS BUT ALSO TASK LEVEL Task lists don’t give you an oversight of how all tests fit together Basic lists of Projects won’t show you what Task is needed next from each person Simple Gantts or Card Views don’t let you move Tasks independently of Projects It is entirely likely (and recommended) to be in planning phase of Test 4 and design phase of Test 3 when Test 1 is live and Test 2 is in Final QA
  55. 55. LIQUID PLANNER IS THIS POWER USER’S CHOICE Ranged estimates for delivery and roll-up, drill down from Client to Project to Task level Each task can have a checklist, deadline, comments, notes, attachments, tags and much more There is also time tracking; a full Resource, Project, Client time analytics suite and an API Zoom out to Year, Quarter and Month, Zoom in to Week and Day views Users see their own task list, but the order they work on them fits the most efficient way for the whole show to move forward.
  56. 56. PLAN TO GET BUSIER – INCREASE CAPACITY AND CAPABILITY The more capable the team is the more (good) work they can do in the same time So increasingly ability increases capacity. But you might need to plan on reducing capacity to create the overhead needed to improve people’s skills If you are short on capacity or capability there are highly experienced professionals out there who can help *cough* www.trsdigital.com *cough*
  57. 57. TESTS WILL BREED TESTS, WINS WILL BREED MORE TESTS Each test should teach you something, even if it is to not test like that again Every metric should point to other areas to explore, user reactions you can investigate Some tests will show great value and prompt many follow ups
  58. 58. ACCEPT CHANGE. MAINTAIN INVESTMENT DON’T CUT CORNERS This is often where successful testing initiatives fall over Initial wins “prove” the original plan was right and the plan becomes fixed Not deviating from the original plan can miss opportunities The other side to that is Costs and Time Initially it may be proof of concept, with a limited budget and resource. But once your Process has shown results cutting back on budget and resource (or leaving it static as your ambition grows) will choke growth Expecting to grow from process efficiency alone is foolish Expecting the same results when you remove support after the initial excitement is unrealistic Investment in time, resource, training and increasing the power given to the roadmap will accelerate success Keeping investment the same but expecting more, will not work.
  59. 59. EXECUTE ON YOUR PLANS BUT WRITE STUFF DOWN! One of the areas many initiatives fail is when the plans change but people don’t remember why New blood in the team, scope creep, unexpected use cases Document every step Use version control Monitor what was the original concept and how it changed to the delivered product Look at commonly occurring issues. Make changes in your process to remove issues and double down on good practice
  60. 60. RECORD YOUR DESIGN CONCEPTS Again looking back in a year your site may well have changed again Again you need to know what “right” looks like when it’s built Changes that come in during the build process are untested, you might run a test that does not fit your hypothesis or metrics Looking at wireframes, mockups and screengrabs of the final QA in your documentation can help explain where you deviated Keep a record of before and after A snapshot of the environment where the change you made was found Don’t assume a “Win” from this year will work the same next year if everything else around it has changed
  61. 61. RECORD YOUR WINS, LOSSES, MISTAKES Again looking back in a year your memory will have faded You are learning lessons in test structure, metric selection, how your site works, how well your process works Document this so you can learn from these Document so your fallible memory is not the only record Document so new team members have a knowledge base they can work from
  62. 62. ADAPT TO CIRCUMSTANCES: SURVIVAL OF THE FITTEST Each test, each planning session, each delayed release, each change to plans should be recorded If you are being a new kind of Hero then you are tracking time and seeing how changes affect the overall plan If you are being smart you regularly look at what burns time, what causes issues in planning, what causes tests to fail QA, what makes for unclear results And each time you learn you revise your plan, look at what tests you have coming that would repeat those mistakes Eliminate those issues, Evolve your plan and your approach Optimise your process, not just your website. If you want to go faster you can add more resource If you want to go faster and get better you need to improve your process You reach a point where just adding more bodies will start to cause problems If you have treated the incremental optimisation of your program as seriously as the optimisation of your site Then more fuel and more power will make you go even faster If you haven’t and you add more fuel and power you can spin out or crash NB I hate the cogs on this Smart Art. That’s not how cogs work, the arrows need to be bi-directional on the big cog or some variable switching pulley on the counter-turning cog for this to even start to make sense
  63. 63. NEARLY DONE Bet you thought this would never end, nearly time for the coffee break We’ll finish with a quick recap
  64. 64. GOOD BOARDING = UNDERSTAND THE PEOPLE
  65. 65. DISCOVERY = UNDERSTAND THE ENVIRONMENT AND EQUIPMENT
  66. 66. DATA DRIVEN DECISIONS, OTHERWISE IT’S GUESSWORK
  67. 67. DECIDE – DON’T JUST COLLECT DATA AND TAKE NO ACTION All too easy to think data and a plan has to be perfect to start All you need is Good Enough The process is to refine all of this as you go along
  68. 68. STRIKE A PRAGMATIC BALANCE THAT SUITS YOUR SITUATION
  69. 69. BE CREATIVE
  70. 70. BALANCE TO SUIT YOUR CAPACITY
  71. 71. MAKE EVERY IDEA EARN ITS PLACE ON THE LIST
  72. 72. EXPLAIN TO STAKEHOLDERS, USE DATA TO BACK UP YOUR POSITION
  73. 73. BY DEFINITION EVERYTHING IS ASAP Because Soon and Possible are subjective variables
  74. 74. DON’T GET CAUGHT UP IN YOUR OWN LEGEND Be “Get things done” rather than “Look at me”
  75. 75. A TESTING PLAN REQUIRES PLANNING Don’t forget Common Sense Common Sense and pragmatism go a long way
  76. 76. WRITE IT DOWN, BE PREPARED TO CHANGE IT
  77. 77. BAD RESULTS ARE STILL RESULTS, LEARN FROM THEM
  78. 78. BE AGILE: SEE AN UNEXPECTED OPPORTUNITY THEN EXPLOIT NEW AREAS
  79. 79. SEX PANTHER, 60% OF THE TIME IT WORKS EVERY TIME Thank you for your time That name again is Mr Plow Tim Stewart from trsdigital
Anúncio