When your testing roadmap includes dozens of ideas (each with unique requirements) and each team member is vying for her idea to be run first, effective prioritization becomes paramount. This session will focus on the considerations, tools and frameworks you can use to make sure your roadmap is appropriately prioritized to meet your goals.
16. Activity: Write down what takes effort to set up a test.
Effort
Teams (or People)
• Power user(s)
• Designer(s)
• Developer(s)
• Graphic designer(s)
• Photography team
• SEO and/or SEM team
• IT department
• QA department
• Executive sponsor
•HTML
•CSS
•Javascript
•Backend changes (reordering
pages in a checkout flow)
•Advanced audience targeting
conditions (cookies, ad
campaigns, custom tags etc.)
Technical
18. Impact
•Higher Conversion Rate
• Purchases
• Sign ups
• Engagement
•More of Something
• Revenue
• Pageviews
•Less of Something
• Internal cost reductions
Hard Impact - Quantifiable
Poll: What is the primary way a test’s impact is measured?
19. Impact
Soft Impact - Meaningful
•Internal Buy-in
• Executive sponsorship
• Excitement (test ideas from
more people)
• Resource allocation
(dedicated developer)
• Access (places testing is
allowed to happen)
•Implementation (winners
actually implemented)
Hard Impact - Quantifiable
•Higher Conversion Rate
• Purchases
• Sign ups
• Engagement
•More of Something
• Revenue
• Pageviews
•Less of Something
• Internal cost reductions
Activity: Write down the primary ways your tests impacts will be
measured?
22. Effort vs. Impact
Test Name Effort Impact
Test 1 Low High
Test 2 Low Medium
Test 3 Low Low
Test 4 Medium High
Test 5 Medium Medium
Test 6 Medium Low
Test 7 High High
Test 8 High Medium
Test 9 High Low
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
Priority
First Second Third
23. Test Name Effort Impact
Test 1 Low High
Test 2 Low Medium
Test 3 Low Low
Test 4 Medium High
Test 5 Medium Medium
Test 6 Medium Low
Test 7 High High
Test 8 High Medium
Test 9 High Low
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
Prioritized by EffortPrioritized by Impact
Effort vs. Impact
Priority
First Second Third
24. Chat: What could be done to
this framework to make it
more useful?
25. Use Numbers Instead of Words & Sum the Scores
Quantify Effort & Impact
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
This impact framework…
26. Use Numbers Instead of Words & Sum the Scores
Quantify Effort & Impact
…becomes this mixed framework.
Priority Test Name Effort Impact Score
1 Test 1 3 3 6
2 Test 2 2 3 5
3 Test 4 3 2 5
4 Test 3 1 3 4
5 Test 5 2 2 4
6 Test 7 3 1 4
7 Test 6 1 2 3
8 Test 8 2 1 3
9 Test 9 1 1 2
Test Name Effort Impact
Test 1 Low High
Test 2 Medium High
Test 3 High High
Test 4 Low Medium
Test 5 Medium Medium
Test 6 High Medium
Test 7 Low Low
Test 8 Medium Low
Test 9 High Low
This impact framework…
30. Create a Classic, Detailed Rubric
Rubric Method #1
Effort Component 3 2 1
HTML No HTML required
HTML required but can
be done within
Optimizely’s visual editor
Detailed, custom HTML
needs to be created
Javascript / jQuery No Javascript required
Simple manipulation of
jQuery in Optimizely’s
“Edit Code” section
Detailed, custom
Javascript or jQuery
needs to be created
Graphic Design
No graphic design
required
Quick graphic design
element needed
Detailed graphic design
work needed
Executive Sponsor
Executive sponsor not
needed
[intentionally blank]
Executive sponsor
approval needed
Example Effort Rubric
=
31. Create a Classic, Detailed Rubric
Rubric Method #1
Impact Component 3 2 1
Conversion Rate
Test has a high likelihood of
directly impacting our
conversion rate.
Test may or may not directly
impact our conversion rate
but should impact a leading
indicator metric.
This test will likely only
impact a leading indicator
metric.
Excitement
This test will generate a lot of
internal excitement for
testing and build momentum.
This test will interest and
satisfy some team members.
This test will only matter to a
few people and get little
visibility regardless of the
outcome.
Implementation
If this test has a winning
variation it will definitely be
implemented immediately.
If this test has a winning
variation it will probably be
implemented or it will take
some time before it is.
Even if this test produces a
winning variation, there is
only a small chance it will be
implemented due to buy-in
or technical constraints.
Example Impact Rubric
=
36. Poll: How can test ideas be
collected in a way that
incorporates rubric scoring…
automatically?
37. Poll: How can test ideas be collected in a way that
incorporates rubric scoring…automatically?
#1
Impossible! It
simply cannot
be done!
38. Poll: How can test ideas be collected in a way that
incorporates rubric scoring…automatically?
#1
Impossible! It
simply cannot
be done!
#2
We’ll create an
exploratory committee
to discover this
method. We should
have an answer when
Tesla has a flying car.
39. Poll: How can test ideas be collected in a way that
incorporates rubric scoring…automatically?
#1
Impossible! It
simply cannot
be done!
#2
We’ll create an
exploratory committee
to discover this
method. We should
have an answer when
Tesla has a flying car.
#3
Create a test idea submission form
that utilizes questions asking what
resources, skill-sets etc. are
required for a given idea. The
responses to this form populate a
spreadsheet with built in formulas
that add or subtract points based
on the responses. Therefore as
soon as an idea is submitted, a
score is generated and the list can
be sorted by scores.
40. Poll: How can test ideas be collected in a way that
incorporates rubric scoring…automatically?
#1
Impossible! It
simply cannot
be done!
#2
We’ll create an
exploratory committee
to discover this
method. We should
have an answer when
Tesla has a flying car.
#3
Create a test idea submission form
that utilizes questions asking what
resources, skill-sets etc. are
required for a given idea. The
responses to this form populate a
spreadsheet with built in formulas
that add or subtract points based
on the responses. Therefore as
soon as an idea is submitted, a
score is generated and the list can
be sorted by scores.
41. So the answer is #3. Now what?
1. Create a test idea
submission form
2. Include questions about
required resources & skill-
sets etc. required.
3. Modify the spreadsheet with
built in formulas that add or
subtract points based on
the responses.
4. Sort the spreadsheet based
on the final prioritization
score.
+
Auto-Prioritization
Recipe
= +
Scoring: Common
Spreadsheet Formulas
Forms: Google Forms
Responses: Google Sheets
Scoring: Common
Spreadsheet Formulas
42. Quick Chat: Name one* tool
you can use to help with
prioritization?
43. Quick Chat: Name one* tool you can use
to help with prioritization?
Forms: Google Forms Responses: Google Sheets
* Google Forms uses Google Sheets automatically.
45. Idea Prioritization Tools
Google Sheets
Automatically integrates
with Google Forms, always
shareable in the cloud
Microsoft Excel
Tried and true with
advanced functionality for
graphing etc., can be
cloud-based
Smartsheet
Set up for seamless project
management and timelines,
cloud-based
Atlassian’s JIRA
Ticketing system
incorporating elements of
form submission tool and a
sortable spreadsheet
47. Poll: Why would a “workflow
management” tool help with
prioritization?
48. Poll: Why would a “workflow management” tool help with
prioritization?
•Choice #1 - A test can’t be the first priority if it needs the expertise
of someone who’s on vacation.
•Choice #2 - A test can’t be the last priority if it’s part of a time-
sensitive marketing campaign.
•Choice #3 - Any test idea submitted by my boss is always the top
priority.
•Choice #4 - Both #1 and #2 are correct but definitely not #3.
49. Workflow & Project Management
Tools
Smartsheet
Set up for seamless project
management and timelines,
cloud-based
Atlassian’s JIRA
Ticketing system
incorporating elements of
form submission tool and a
sortable spreadsheet, also
can utilize “cards” that
track the status of each
step
Trello JIRA
One of the most popular
testing workflow
management tools,
incorporates “cards”, to-do
lists and is one of the more
visual systems out there
52. Additional Resources
Optiverse contains content on a variety of topics in a
variety of formats including our online user community,
knowledge base, and learning academy!
Community Knowledge Base Learning Academy