When your testing roadmap includes dozens of ideas (each with unique requirements) and each team member is vying for her idea to be run first, effective prioritization becomes paramount. This session will focus on the considerations, tools and frameworks you can use to make sure your roadmap is appropriately prioritized to meet your goals.

1. Prioritize Your Roadmap
Strategy Consultant: Ryan Lillis
August 12th, 2015
An Optimizely Online Workshop

2. Introduction
Ryan Lillis
Strategy Consultant
ryan.lillis@optimizely.com

3. Today’s Takeaways

4. Today’s Takeaways
#1
Why Prioritize &
Key Components
#2
Tools to Prioritize
& Effective Usage

5. Today’s Tools

6. Today’s Tools
#1
Your List of Test
Ideas

7. Today’s Tools
#1
Your List of Test
Ideas
#2
A Blank
Spreadsheet

8. 1. Prioritization Components
2. Components in Action
3. Tools for Organization & Prioritization
Agenda
Key Focus Areas

9. Three Questions

10. #1 - Poll: Do you have a list
of test ideas?

11. #2 - Poll: How many tests
are in your list?

12. #3 - Poll: Do you have a
prioritization framework and
is it followed?

13. Prioritization
Components

14. Effort Impact

15. Effort
Activity: Write down what takes effort to set up a test.

16. Activity: Write down what takes effort to set up a test.
Effort
Teams (or People)
• Power user(s)
• Designer(s)
• Developer(s)
• Graphic designer(s)
• Photography team
• SEO and/or SEM team
• IT department
• QA department
• Executive sponsor
•HTML
•CSS
•Javascript
•Backend changes (reordering
pages in a checkout flow)
•Advanced audience targeting
conditions (cookies, ad
campaigns, custom tags etc.)
Technical

17. Impact
Poll: What is the primary way a test’s impact is measured?

18. Impact
•Higher Conversion Rate
• Purchases
• Sign ups
• Engagement
•More of Something
• Revenue
• Pageviews
•Less of Something
• Internal cost reductions
Hard Impact - Quantifiable
Poll: What is the primary way a test’s impact is measured?

19. Impact
Soft Impact - Meaningful
•Internal Buy-in
• Executive sponsorship
• Excitement (test ideas from
more people)
• Resource allocation
(dedicated developer)
• Access (places testing is
allowed to happen)
•Implementation (winners
actually implemented)
Hard Impact - Quantifiable
•Higher Conversion Rate
• Purchases
• Sign ups
• Engagement
•More of Something
• Revenue
• Pageviews
•Less of Something
• Internal cost reductions
Activity: Write down the primary ways your tests impacts will be
measured?

20. Components
In Action

21. Basic Framework
Effort vs. Impact

22. Effort vs. Impact
Test Name Effort Impact
Test 1 Low High
Test 2 Low Medium
Test 3 Low Low
Test 4 Medium High
Test 5 Medium Medium
Test 6 Medium Low
Test 7 High High
Test 8 High Medium
Test 9 High Low
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
Priority
First Second Third

23. Test Name Effort Impact
Test 1 Low High
Test 2 Low Medium
Test 3 Low Low
Test 4 Medium High
Test 5 Medium Medium
Test 6 Medium Low
Test 7 High High
Test 8 High Medium
Test 9 High Low
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
Prioritized by EffortPrioritized by Impact
Effort vs. Impact
Priority
First Second Third

24. Chat: What could be done to
this framework to make it
more useful?

25. Use Numbers Instead of Words & Sum the Scores
Quantify Effort & Impact
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
This impact framework…

26. Use Numbers Instead of Words & Sum the Scores
Quantify Effort & Impact
…becomes this mixed framework.
Priority Test Name Effort Impact Score
1 Test 1 3 3 6
2 Test 2 2 3 5
3 Test 4 3 2 5
4 Test 3 1 3 4
5 Test 5 2 2 4
6 Test 7 3 1 4
7 Test 6 1 2 3
8 Test 8 2 1 3
9 Test 9 1 1 2
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
This impact framework…

27. Poll: Who’s satisfied with
this framework?

28. Consider Your Key Components & Create a Rubric
Objectively Quantify Effort & Impact

29. Create a Classic, Detailed Rubric
Rubric Method #1

30. Create a Classic, Detailed Rubric
Rubric Method #1
Effort Component 3 2 1
HTML No HTML required
HTML required but can
be done within
Optimizely’s visual editor
Detailed, custom HTML
needs to be created
Javascript / jQuery No Javascript required
Simple manipulation of
jQuery in Optimizely’s
“Edit Code” section
Detailed, custom
Javascript or jQuery
needs to be created
Graphic Design
No graphic design
required
Quick graphic design
element needed
Detailed graphic design
work needed
Executive Sponsor
Executive sponsor not
needed
[intentionally blank]
Executive sponsor
approval needed
Example Effort Rubric
=

31. Create a Classic, Detailed Rubric
Rubric Method #1
Impact Component 3 2 1
Conversion Rate
Test has a high likelihood of
directly impacting our
conversion rate.
Test may or may not directly
impact our conversion rate
but should impact a leading
indicator metric.
This test will likely only
impact a leading indicator
metric.
Excitement
This test will generate a lot of
internal excitement for
testing and build momentum.
This test will interest and
satisfy some team members.
This test will only matter to a
few people and get little
visibility regardless of the
outcome.
Implementation
If this test has a winning
variation it will definitely be
implemented immediately.
If this test has a winning
variation it will probably be
implemented or it will take
some time before it is.
Even if this test produces a
winning variation, there is
only a small chance it will be
implemented due to buy-in
or technical constraints.
Example Impact Rubric
=

32. Example Spreadsheet
Let’s Look at an Example of What a
Prioritized Spreadsheet May Look
Like

33. Tools for
Organization &
Prioritization

34. Three Tool Categories
Idea Prioritization Workflow ManagementIdea Collection

35. Three Tool Categories
Idea Prioritization Workflow ManagementIdea Collection

36. Poll: How can test ideas be
collected in a way that
incorporates rubric scoring…
automatically?

37. Poll: How can test ideas be collected in a way that
incorporates rubric scoring…automatically?
#1
Impossible! It
simply cannot
be done!

38. Poll: How can test ideas be collected in a way that
incorporates rubric scoring…automatically?
#1
Impossible! It
simply cannot
be done!
#2
We’ll create an
exploratory committee
to discover this
method. We should
have an answer when
Tesla has a flying car.

39. Poll: How can test ideas be collected in a way that
incorporates rubric scoring…automatically?
#1
Impossible! It
simply cannot
be done!
#2
We’ll create an
exploratory committee
to discover this
method. We should
have an answer when
Tesla has a flying car.
#3
Create a test idea submission form
that utilizes questions asking what
resources, skill-sets etc. are
required for a given idea. The
responses to this form populate a
spreadsheet with built in formulas
that add or subtract points based
on the responses. Therefore as
soon as an idea is submitted, a
score is generated and the list can
be sorted by scores.

40. Poll: How can test ideas be collected in a way that
incorporates rubric scoring…automatically?
#1
Impossible! It
simply cannot
be done!
#2
We’ll create an
exploratory committee
to discover this
method. We should
have an answer when
Tesla has a flying car.
#3
Create a test idea submission form
that utilizes questions asking what
resources, skill-sets etc. are
required for a given idea. The
responses to this form populate a
spreadsheet with built in formulas
that add or subtract points based
on the responses. Therefore as
soon as an idea is submitted, a
score is generated and the list can
be sorted by scores.

41. So the answer is #3. Now what?
1. Create a test idea
submission form
2. Include questions about
required resources & skill-
sets etc. required.
3. Modify the spreadsheet with
built in formulas that add or
subtract points based on
the responses.
4. Sort the spreadsheet based
on the final prioritization
score.
+
Auto-Prioritization
Recipe
= +
Scoring: Common
Spreadsheet Formulas
Forms: Google Forms
Responses: Google Sheets
Scoring: Common
Spreadsheet Formulas

42. Quick Chat: Name one* tool
you can use to help with
prioritization?

43. Quick Chat: Name one* tool you can use
to help with prioritization?
Forms: Google Forms Responses: Google Sheets
* Google Forms uses Google Sheets automatically.

44. Three Tool Categories
Idea Prioritization Workflow ManagementIdea Collection

45. Idea Prioritization Tools
Google Sheets
Automatically integrates
with Google Forms, always
shareable in the cloud
Microsoft Excel
Tried and true with
advanced functionality for
graphing etc., can be
cloud-based
Smartsheet
Set up for seamless project
management and timelines,
cloud-based
Atlassian’s JIRA
Ticketing system
incorporating elements of
form submission tool and a
sortable spreadsheet

46. Three Tool Categories
Idea Prioritization Workflow ManagementIdea Collection

47. Poll: Why would a “workflow
management” tool help with
prioritization?

48. Poll: Why would a “workflow management” tool help with
prioritization?
•Choice #1 - A test can’t be the first priority if it needs the expertise
of someone who’s on vacation.
•Choice #2 - A test can’t be the last priority if it’s part of a time-
sensitive marketing campaign.
•Choice #3 - Any test idea submitted by my boss is always the top
priority.
•Choice #4 - Both #1 and #2 are correct but definitely not #3.

49. Workflow & Project Management
Tools
Smartsheet
Set up for seamless project
management and timelines,
cloud-based
Atlassian’s JIRA
Ticketing system
incorporating elements of
form submission tool and a
sortable spreadsheet, also
can utilize “cards” that
track the status of each
step
Trello JIRA
One of the most popular
testing workflow
management tools,
incorporates “cards”, to-do
lists and is one of the more
visual systems out there

50. Recap

51. 1. Prioritization Components
2. Components in Action
3. Tools for Organization & Prioritization
Original Agenda

52. Additional Resources
Optiverse contains content on a variety of topics in a
variety of formats including our online user community,
knowledge base, and learning academy!
Community Knowledge Base Learning Academy

53. Thank You For Attending!