Prioritize Your Roadmap Strategy Consultant:
Ryan Lillis August 12th, 2015 An Optimizely Online Workshop

Introduction Ryan Lillis Strategy Consultant
ryan.lillis@optimizely.com

Today’s Takeaways

Today’s Takeaways #1 Why Prioritize
& Key Components #2 Tools to Prioritize & Effective Usage

Today’s Tools

Today’s Tools #1 Your List
of Test Ideas

Today’s Tools #1 Your List
of Test Ideas #2 A Blank Spreadsheet

1. Prioritization Components 2. Components
in Action 3. Tools for Organization & Prioritization Agenda Key Focus Areas

Three Questions

#1 - Poll: Do you
have a list of test ideas?

#2 - Poll: How many
tests are in your list?

#3 - Poll: Do you
have a prioritization framework and is it followed?

Prioritization
Components

Effort Impact

Effort Activity: Write down what
takes effort to set up a test.

Activity: Write down what takes
effort to set up a test. Effort Teams (or People) • Power user(s) • Designer(s) • Developer(s) • Graphic designer(s) • Photography team • SEO and/or SEM team • IT department • QA department • Executive sponsor •HTML •CSS •Javascript •Backend changes (reordering pages in a checkout flow) •Advanced audience targeting conditions (cookies, ad campaigns, custom tags etc.) Technical

Impact Poll: What is the
primary way a test’s impact is measured?

Impact •Higher Conversion Rate •
Purchases • Sign ups • Engagement •More of Something • Revenue • Pageviews •Less of Something • Internal cost reductions Hard Impact - Quantifiable Poll: What is the primary way a test’s impact is measured?

Impact Soft Impact - Meaningful
•Internal Buy-in • Executive sponsorship • Excitement (test ideas from more people) • Resource allocation (dedicated developer) • Access (places testing is allowed to happen) •Implementation (winners actually implemented) Hard Impact - Quantifiable •Higher Conversion Rate • Purchases • Sign ups • Engagement •More of Something • Revenue • Pageviews •Less of Something • Internal cost reductions Activity: Write down the primary ways your tests impacts will be measured?

Components
In Action

Basic Framework
Effort vs. Impact

Effort vs. Impact Test Name
Effort Impact Test 1 Low High Test 2 Low Medium Test 3 Low Low Test 4 Medium High Test 5 Medium Medium Test 6 Medium Low Test 7 High High Test 8 High Medium Test 9 High Low Test Name Effort Impact Test 1 Low High Test 2 Medium High Test 3 High High Test 4 Low Medium Test 5 Medium Medium Test 6 High Medium Test 7 Low Low Test 8 Medium Low Test 9 High Low Priority First Second Third

Test Name Effort Impact Test
1 Low High Test 2 Low Medium Test 3 Low Low Test 4 Medium High Test 5 Medium Medium Test 6 Medium Low Test 7 High High Test 8 High Medium Test 9 High Low Test Name Effort Impact Test 1 Low High Test 2 Medium High Test 3 High High Test 4 Low Medium Test 5 Medium Medium Test 6 High Medium Test 7 Low Low Test 8 Medium Low Test 9 High Low Prioritized by EffortPrioritized by Impact Effort vs. Impact Priority First Second Third

Chat: What could be done
to this framework to make it more useful?

Use Numbers Instead of Words
& Sum the Scores Quantify Effort & Impact Test Name Effort Impact Test 1 Low High Test 2 Medium High Test 3 High High Test 4 Low Medium Test 5 Medium Medium Test 6 High Medium Test 7 Low Low Test 8 Medium Low Test 9 High Low This impact framework…

Use Numbers Instead of Words
& Sum the Scores Quantify Effort & Impact …becomes this mixed framework. Priority Test Name Effort Impact Score 1 Test 1 3 3 6 2 Test 2 2 3 5 3 Test 4 3 2 5 4 Test 3 1 3 4 5 Test 5 2 2 4 6 Test 7 3 1 4 7 Test 6 1 2 3 8 Test 8 2 1 3 9 Test 9 1 1 2 Test Name Effort Impact Test 1 Low High Test 2 Medium High Test 3 High High Test 4 Low Medium Test 5 Medium Medium Test 6 High Medium Test 7 Low Low Test 8 Medium Low Test 9 High Low This impact framework…

Poll: Who’s satisfied with this
framework?

Consider Your Key Components &
Create a Rubric Objectively Quantify Effort & Impact

Create a Classic, Detailed Rubric
Rubric Method #1

Create a Classic, Detailed Rubric
Rubric Method #1 Effort Component 3 2 1 HTML No HTML required HTML required but can be done within Optimizely’s visual editor Detailed, custom HTML needs to be created Javascript / jQuery No Javascript required Simple manipulation of jQuery in Optimizely’s “Edit Code” section Detailed, custom Javascript or jQuery needs to be created Graphic Design No graphic design required Quick graphic design element needed Detailed graphic design work needed Executive Sponsor Executive sponsor not needed [intentionally blank] Executive sponsor approval needed Example Effort Rubric =

Create a Classic, Detailed Rubric
Rubric Method #1 Impact Component 3 2 1 Conversion Rate Test has a high likelihood of directly impacting our conversion rate. Test may or may not directly impact our conversion rate but should impact a leading indicator metric. This test will likely only impact a leading indicator metric. Excitement This test will generate a lot of internal excitement for testing and build momentum. This test will interest and satisfy some team members. This test will only matter to a few people and get little visibility regardless of the outcome. Implementation If this test has a winning variation it will definitely be implemented immediately. If this test has a winning variation it will probably be implemented or it will take some time before it is. Even if this test produces a winning variation, there is only a small chance it will be implemented due to buy-in or technical constraints. Example Impact Rubric =

Example Spreadsheet Let’s Look at
an Example of What a Prioritized Spreadsheet May Look Like

Tools for
Organization &
Prioritization

Three Tool Categories Idea Prioritization
Workflow ManagementIdea Collection

Three Tool Categories Idea Prioritization
Workflow ManagementIdea Collection

Poll: How can test ideas
be collected in a way that incorporates rubric scoring… automatically?

Poll: How can test ideas
be collected in a way that incorporates rubric scoring…automatically? #1 Impossible! It simply cannot be done!

Poll: How can test ideas
be collected in a way that incorporates rubric scoring…automatically? #1 Impossible! It simply cannot be done! #2 We’ll create an exploratory committee to discover this method. We should have an answer when Tesla has a flying car.

Poll: How can test ideas
be collected in a way that incorporates rubric scoring…automatically? #1 Impossible! It simply cannot be done! #2 We’ll create an exploratory committee to discover this method. We should have an answer when Tesla has a flying car. #3 Create a test idea submission form that utilizes questions asking what resources, skill-sets etc. are required for a given idea. The responses to this form populate a spreadsheet with built in formulas that add or subtract points based on the responses. Therefore as soon as an idea is submitted, a score is generated and the list can be sorted by scores.

Poll: How can test ideas
be collected in a way that incorporates rubric scoring…automatically? #1 Impossible! It simply cannot be done! #2 We’ll create an exploratory committee to discover this method. We should have an answer when Tesla has a flying car. #3 Create a test idea submission form that utilizes questions asking what resources, skill-sets etc. are required for a given idea. The responses to this form populate a spreadsheet with built in formulas that add or subtract points based on the responses. Therefore as soon as an idea is submitted, a score is generated and the list can be sorted by scores.

So the answer is #3.
Now what? 1. Create a test idea submission form 2. Include questions about required resources & skill- sets etc. required. 3. Modify the spreadsheet with built in formulas that add or subtract points based on the responses. 4. Sort the spreadsheet based on the final prioritization score. + Auto-Prioritization Recipe = + Scoring: Common Spreadsheet Formulas Forms: Google Forms Responses: Google Sheets Scoring: Common Spreadsheet Formulas

Quick Chat: Name one* tool
you can use to help with prioritization?

Quick Chat: Name one* tool
you can use to help with prioritization? Forms: Google Forms Responses: Google Sheets * Google Forms uses Google Sheets automatically.

Three Tool Categories Idea Prioritization
Workflow ManagementIdea Collection

Idea Prioritization Tools Google Sheets
Automatically integrates with Google Forms, always shareable in the cloud Microsoft Excel Tried and true with advanced functionality for graphing etc., can be cloud-based Smartsheet Set up for seamless project management and timelines, cloud-based Atlassian’s JIRA Ticketing system incorporating elements of form submission tool and a sortable spreadsheet

Three Tool Categories Idea Prioritization
Workflow ManagementIdea Collection

Poll: Why would a “workflow
management” tool help with prioritization?

Poll: Why would a “workflow
management” tool help with prioritization? •Choice #1 - A test can’t be the first priority if it needs the expertise of someone who’s on vacation. •Choice #2 - A test can’t be the last priority if it’s part of a time- sensitive marketing campaign. •Choice #3 - Any test idea submitted by my boss is always the top priority. •Choice #4 - Both #1 and #2 are correct but definitely not #3.

Workflow & Project Management Tools
Smartsheet Set up for seamless project management and timelines, cloud-based Atlassian’s JIRA Ticketing system incorporating elements of form submission tool and a sortable spreadsheet, also can utilize “cards” that track the status of each step Trello JIRA One of the most popular testing workflow management tools, incorporates “cards”, to-do lists and is one of the more visual systems out there

Recap

1. Prioritization Components 2. Components
in Action 3. Tools for Organization & Prioritization Original Agenda

Additional Resources Optiverse contains content
on a variety of topics in a variety of formats including our online user community, knowledge base, and learning academy! Community Knowledge Base Learning Academy

Thank You For Attending!