SlideShare uma empresa Scribd logo
1 de 64
Baixar para ler offline
MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112
A/B Testing
HOW TO BREAK THINGS AND FAIL FAST…
WITHOUT BREAKING THINGS
About Atlassian
• Products include: JIRA, Confluence, Bitbucket, HipChat…
• Founded in 2002
• Over 35,000 Customers
• Over 700 employees globally
• No sales people
!
!
!
We h e l p g r e a t t e a m s b u i l d b e t t e r s o f t w a r e , t o g e t h e r.
Think big, act small, fail fast, learn rapidly.
"LEAN SOFTWARE DEVELOPMENT” BY MARY AND TOM POPPENDIECK (2003)
”“
Atlassian A/B Testing is cross functional
• Email
• Product Management
• Customer Platform
• Customer Advocates
• Internal Systems
!
!
!
C e n t r a l G r o w t h H a c k i n g Te a m t o s u p p o r t :
• Leads
• Product Marketing
• Support
• Bitbucket
• HipChat
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Where do experiment ideas come
from?
Anywhere.
What are you are trying to
improve?
Blimp Dashboard
JIRA – Ticket
• Pebbles
• Change in copy or CTA presentation
• Rocks
• New page (e.g. no Dev Tools family page)
• Boulders
• Simplified order form.
!
!
!
Types of tests
Examples
Reduce Friction
Increase Visibility
Test drive campaigns
Simplify
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
What are you are trying to
improve?
Blimp Dashboard
JIRA – Prioritized Backlog
JIRA – Ticket
JIRA – Ticket
Experiment Illuminati
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Build
Optimizely – Build
Prepare for results.
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
Confluence – Corresponding Test Page
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
JIRA – Prioritized Backlog
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Start Experiment
Optimizely – Project Code
Magnolia CMS – Include Optimizely Checkbox
Magnolia CMS – Include Optimizely Checkbox
Confluence – Corresponding Test Page
SCHEDULE TO AVOID CONFLICTS
BUILD OUT GOOD CANDIDATES
TRIAGE BASED ON POTENTIAL IMPACT
COLLECT GREAT IDEAS
Our Process
ANALYZE RESULTS
RUN EXPERIMENTS
Optimizely – Results
What are you are trying to
improve?
Blimp Dashboard
Optimizely – Results
Confluence – Corresponding Test Page
Confluence – Results Archive
• Ideas are everywhere
• Everything affecting conversion is tested
• No traffic wasted
• Be aggressive
• Regular cadence
• Never run a test that you wouldn’t want to win
Our Test Philosophy
• Ideas are everywhere collect as many as you can, prioritize based on
impact.
• Use ticket number everywhere so everyone is referring to the same
test.
• Involve stakeholders and developers… and get approvals.
Key Take Aways
Thank you!
MARK HALVORSON • HEAD OF INTERACTIVE • ATLASSIAN SOFTWARE • @halv0112
A/B Testing - How to Break Things and Fail Fast... Without Breaking Things

Mais conteúdo relacionado

Mais procurados

The 6 Mindsets of Red Ocean Disruption Teams: Tools for Rapidly Discovering a...
The 6 Mindsets of Red Ocean Disruption Teams: Tools for Rapidly Discovering a...The 6 Mindsets of Red Ocean Disruption Teams: Tools for Rapidly Discovering a...
The 6 Mindsets of Red Ocean Disruption Teams: Tools for Rapidly Discovering a...
Rod King, Ph.D.
 

Mais procurados (20)

20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them20 top AB testing mistakes and how to avoid them
20 top AB testing mistakes and how to avoid them
 
Embracing Change in a Startup - Responsive Operating Model
Embracing Change in a Startup - Responsive Operating ModelEmbracing Change in a Startup - Responsive Operating Model
Embracing Change in a Startup - Responsive Operating Model
 
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests#Measurefest : 20 Simple Ways to Fuck Up your AB tests
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
 
Why do my AB tests suck? measurecamp
Why do my AB tests suck?   measurecampWhy do my AB tests suck?   measurecamp
Why do my AB tests suck? measurecamp
 
Agile vs agile (vs agile)
Agile vs agile (vs agile)Agile vs agile (vs agile)
Agile vs agile (vs agile)
 
Advanced Scrum: Answering the Difficult Questions
Advanced Scrum: Answering the Difficult QuestionsAdvanced Scrum: Answering the Difficult Questions
Advanced Scrum: Answering the Difficult Questions
 
Dc20 james birnie - agile is a dirty word
Dc20   james birnie - agile is a dirty wordDc20   james birnie - agile is a dirty word
Dc20 james birnie - agile is a dirty word
 
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to MeBrighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to meDigital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
 
The 6 Mindsets of Red Ocean Disruption Teams: Tools for Rapidly Discovering a...
The 6 Mindsets of Red Ocean Disruption Teams: Tools for Rapidly Discovering a...The 6 Mindsets of Red Ocean Disruption Teams: Tools for Rapidly Discovering a...
The 6 Mindsets of Red Ocean Disruption Teams: Tools for Rapidly Discovering a...
 
Interact London - 21 Oct 2015 - Scaling Stupidity
Interact London - 21 Oct 2015 - Scaling StupidityInteract London - 21 Oct 2015 - Scaling Stupidity
Interact London - 21 Oct 2015 - Scaling Stupidity
 
Modern Agile - Joshua Kerievsky
Modern Agile - Joshua KerievskyModern Agile - Joshua Kerievsky
Modern Agile - Joshua Kerievsky
 
Building on the Shoulders of Giants: the Story of Bitbucket Pipelines
Building on the Shoulders of Giants: the Story of Bitbucket PipelinesBuilding on the Shoulders of Giants: the Story of Bitbucket Pipelines
Building on the Shoulders of Giants: the Story of Bitbucket Pipelines
 
LKCE16 - Servant Leadership un-neutered by Mike Burrows
LKCE16 - Servant Leadership un-neutered by Mike BurrowsLKCE16 - Servant Leadership un-neutered by Mike Burrows
LKCE16 - Servant Leadership un-neutered by Mike Burrows
 
Product Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to dieProduct Design is Poo - And we're all going to die
Product Design is Poo - And we're all going to die
 
Agile Anti-patterns
Agile Anti-patternsAgile Anti-patterns
Agile Anti-patterns
 
Coaching Anti-Pattens and common smells
 Coaching Anti-Pattens and common smells Coaching Anti-Pattens and common smells
Coaching Anti-Pattens and common smells
 
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
 
Myths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split TestingMyths, Lies and Illusions of AB and Split Testing
Myths, Lies and Illusions of AB and Split Testing
 

Destaque

Destaque (7)

Go! Go! Gadgets. Writing an OpenSocial Application
Go! Go! Gadgets.  Writing an OpenSocial ApplicationGo! Go! Gadgets.  Writing an OpenSocial Application
Go! Go! Gadgets. Writing an OpenSocial Application
 
Bizsnatch! Why funny is good business.
Bizsnatch!  Why funny is good business.Bizsnatch!  Why funny is good business.
Bizsnatch! Why funny is good business.
 
How to Build Interactive Training that Scales - Magnolia CMS Conference 2013
How to Build Interactive Training that Scales - Magnolia CMS Conference 2013How to Build Interactive Training that Scales - Magnolia CMS Conference 2013
How to Build Interactive Training that Scales - Magnolia CMS Conference 2013
 
SXSW Funworkshop, Let's fix sucky stuff at SXSW, together.
SXSW Funworkshop, Let's fix sucky stuff at SXSW, together.SXSW Funworkshop, Let's fix sucky stuff at SXSW, together.
SXSW Funworkshop, Let's fix sucky stuff at SXSW, together.
 
Web 2.0 at 60mph
Web 2.0 at 60mphWeb 2.0 at 60mph
Web 2.0 at 60mph
 
How to build a great coding culture
How to build a great coding cultureHow to build a great coding culture
How to build a great coding culture
 
Study: The Future of VR, AR and Self-Driving Cars
Study: The Future of VR, AR and Self-Driving CarsStudy: The Future of VR, AR and Self-Driving Cars
Study: The Future of VR, AR and Self-Driving Cars
 

Semelhante a A/B Testing - How to Break Things and Fail Fast... Without Breaking Things

15dominodatasciencepopupseattleseanmcclureslides 151013133441-lva1-app6891
15dominodatasciencepopupseattleseanmcclureslides 151013133441-lva1-app689115dominodatasciencepopupseattleseanmcclureslides 151013133441-lva1-app6891
15dominodatasciencepopupseattleseanmcclureslides 151013133441-lva1-app6891
Sean McClure, Ph.D.
 

Semelhante a A/B Testing - How to Break Things and Fail Fast... Without Breaking Things (20)

Optimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - AtlassianOptimizely Experience Customer Story - Atlassian
Optimizely Experience Customer Story - Atlassian
 
Recruiting great developers
Recruiting great developersRecruiting great developers
Recruiting great developers
 
Keep It Lean - TCUK 2015
Keep It Lean - TCUK 2015Keep It Lean - TCUK 2015
Keep It Lean - TCUK 2015
 
2016 04-07 key note -agile organizations
2016 04-07 key note -agile organizations2016 04-07 key note -agile organizations
2016 04-07 key note -agile organizations
 
Agile frameworks: the why, how and what
Agile frameworks: the why, how and whatAgile frameworks: the why, how and what
Agile frameworks: the why, how and what
 
Leanstartuptoolsforproductowners
LeanstartuptoolsforproductownersLeanstartuptoolsforproductowners
Leanstartuptoolsforproductowners
 
Lean Startup Tools for Scrum Product Owners
Lean Startup Tools for Scrum Product OwnersLean Startup Tools for Scrum Product Owners
Lean Startup Tools for Scrum Product Owners
 
Do Agile Right - Lessons Learned from an Atlassian Product Manager - Sherif M...
Do Agile Right - Lessons Learned from an Atlassian Product Manager - Sherif M...Do Agile Right - Lessons Learned from an Atlassian Product Manager - Sherif M...
Do Agile Right - Lessons Learned from an Atlassian Product Manager - Sherif M...
 
Hanno Jarvet - The Lean Toolkit – Value Stream Mapping and Problem Solving
Hanno Jarvet - The Lean Toolkit – Value Stream Mapping and Problem SolvingHanno Jarvet - The Lean Toolkit – Value Stream Mapping and Problem Solving
Hanno Jarvet - The Lean Toolkit – Value Stream Mapping and Problem Solving
 
Hadoop and the Relational Database: The Best of Both Worlds
Hadoop and the Relational Database: The Best of Both WorldsHadoop and the Relational Database: The Best of Both Worlds
Hadoop and the Relational Database: The Best of Both Worlds
 
How to Power Continuous Deployment at Any Scale with Atlassian
 How to Power Continuous Deployment at Any Scale with Atlassian How to Power Continuous Deployment at Any Scale with Atlassian
How to Power Continuous Deployment at Any Scale with Atlassian
 
Inbound Growth for SaaS Scale-Ups #INBOUND18
Inbound Growth for SaaS Scale-Ups #INBOUND18Inbound Growth for SaaS Scale-Ups #INBOUND18
Inbound Growth for SaaS Scale-Ups #INBOUND18
 
Why agile is failing in large enterprises
Why agile is failing in large enterprisesWhy agile is failing in large enterprises
Why agile is failing in large enterprises
 
How Data Science Builds Better Products - Data Science Pop-up Seattle
How Data Science Builds Better Products - Data Science Pop-up SeattleHow Data Science Builds Better Products - Data Science Pop-up Seattle
How Data Science Builds Better Products - Data Science Pop-up Seattle
 
15dominodatasciencepopupseattleseanmcclureslides 151013133441-lva1-app6891
15dominodatasciencepopupseattleseanmcclureslides 151013133441-lva1-app689115dominodatasciencepopupseattleseanmcclureslides 151013133441-lva1-app6891
15dominodatasciencepopupseattleseanmcclureslides 151013133441-lva1-app6891
 
Hanno Jarvet - VSM, Planning and Problem Solving - ConFu
Hanno Jarvet - VSM, Planning and Problem Solving - ConFuHanno Jarvet - VSM, Planning and Problem Solving - ConFu
Hanno Jarvet - VSM, Planning and Problem Solving - ConFu
 
Biz barcelona 061511
Biz barcelona 061511Biz barcelona 061511
Biz barcelona 061511
 
Biz barcelona 061511
Biz barcelona 061511Biz barcelona 061511
Biz barcelona 061511
 
Growth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - AntwerpGrowth Hacking Conference '17 - Antwerp
Growth Hacking Conference '17 - Antwerp
 
SpatzAI Empowering Bold Idea Sharing, Spat by Spat
SpatzAI Empowering Bold Idea Sharing, Spat by SpatSpatzAI Empowering Bold Idea Sharing, Spat by Spat
SpatzAI Empowering Bold Idea Sharing, Spat by Spat
 

Último

Último (20)

How consumers use technology and the impacts on their lives
How consumers use technology and the impacts on their livesHow consumers use technology and the impacts on their lives
How consumers use technology and the impacts on their lives
 
TAM_AdEx-Cross_Media_Report-Banking_Finance_Investment_(BFSI)_2023.pdf
TAM_AdEx-Cross_Media_Report-Banking_Finance_Investment_(BFSI)_2023.pdfTAM_AdEx-Cross_Media_Report-Banking_Finance_Investment_(BFSI)_2023.pdf
TAM_AdEx-Cross_Media_Report-Banking_Finance_Investment_(BFSI)_2023.pdf
 
20180928 Hofstede Insights Conference Milan The Power of Culture Led Brands.pptx
20180928 Hofstede Insights Conference Milan The Power of Culture Led Brands.pptx20180928 Hofstede Insights Conference Milan The Power of Culture Led Brands.pptx
20180928 Hofstede Insights Conference Milan The Power of Culture Led Brands.pptx
 
W.H.Bender Quote 61 -Influential restaurant and food service industry network...
W.H.Bender Quote 61 -Influential restaurant and food service industry network...W.H.Bender Quote 61 -Influential restaurant and food service industry network...
W.H.Bender Quote 61 -Influential restaurant and food service industry network...
 
Discover Ardency Elite: Elevate Your Lifestyle
Discover Ardency Elite: Elevate Your LifestyleDiscover Ardency Elite: Elevate Your Lifestyle
Discover Ardency Elite: Elevate Your Lifestyle
 
Alpha Media March 2024 Buyers Guide.pptx
Alpha Media March 2024 Buyers Guide.pptxAlpha Media March 2024 Buyers Guide.pptx
Alpha Media March 2024 Buyers Guide.pptx
 
Social media, ppt. Features, characteristics
Social media, ppt. Features, characteristicsSocial media, ppt. Features, characteristics
Social media, ppt. Features, characteristics
 
[Expert Panel] New Google Shopping Ads Strategies Uncovered
[Expert Panel] New Google Shopping Ads Strategies Uncovered[Expert Panel] New Google Shopping Ads Strategies Uncovered
[Expert Panel] New Google Shopping Ads Strategies Uncovered
 
The+State+of+Careers+In+Retention+Marketing-2.pdf
The+State+of+Careers+In+Retention+Marketing-2.pdfThe+State+of+Careers+In+Retention+Marketing-2.pdf
The+State+of+Careers+In+Retention+Marketing-2.pdf
 
Choosing the Right White Label SEO Services to Boost Your Agency's Growth.pdf
Choosing the Right White Label SEO Services to Boost Your Agency's Growth.pdfChoosing the Right White Label SEO Services to Boost Your Agency's Growth.pdf
Choosing the Right White Label SEO Services to Boost Your Agency's Growth.pdf
 
What is Google Search Console and What is it provide?
What is Google Search Console and What is it provide?What is Google Search Console and What is it provide?
What is Google Search Console and What is it provide?
 
2024 Social Trends Report V4 from Later.com
2024 Social Trends Report V4 from Later.com2024 Social Trends Report V4 from Later.com
2024 Social Trends Report V4 from Later.com
 
Digital-Marketing-Into-by-Zoraiz-Ahmad.pptx
Digital-Marketing-Into-by-Zoraiz-Ahmad.pptxDigital-Marketing-Into-by-Zoraiz-Ahmad.pptx
Digital-Marketing-Into-by-Zoraiz-Ahmad.pptx
 
10 Email Marketing Best Practices to Increase Engagements, CTR, And ROI
10 Email Marketing Best Practices to Increase Engagements, CTR, And ROI10 Email Marketing Best Practices to Increase Engagements, CTR, And ROI
10 Email Marketing Best Practices to Increase Engagements, CTR, And ROI
 
Elevate Your Advertising Game: Introducing Billion Broadcaster Lift Advertising
Elevate Your Advertising Game: Introducing Billion Broadcaster Lift AdvertisingElevate Your Advertising Game: Introducing Billion Broadcaster Lift Advertising
Elevate Your Advertising Game: Introducing Billion Broadcaster Lift Advertising
 
Five Essential Tools for International SEO - Natalia Witczyk - SearchNorwich 15
Five Essential Tools for International SEO - Natalia Witczyk - SearchNorwich 15Five Essential Tools for International SEO - Natalia Witczyk - SearchNorwich 15
Five Essential Tools for International SEO - Natalia Witczyk - SearchNorwich 15
 
Enhancing Business Visibility PR Firms in San Francisco
Enhancing Business Visibility PR Firms in San FranciscoEnhancing Business Visibility PR Firms in San Francisco
Enhancing Business Visibility PR Firms in San Francisco
 
personal branding kit for music business
personal branding kit for music businesspersonal branding kit for music business
personal branding kit for music business
 
The Science of Landing Page Messaging.pdf
The Science of Landing Page Messaging.pdfThe Science of Landing Page Messaging.pdf
The Science of Landing Page Messaging.pdf
 
SP Search Term Data Optimization Template.pdf
SP Search Term Data Optimization Template.pdfSP Search Term Data Optimization Template.pdf
SP Search Term Data Optimization Template.pdf
 

A/B Testing - How to Break Things and Fail Fast... Without Breaking Things