A/B Testing: You Might be Driving in the Wrong Direction

•

29 gostaram•24,379 visualizações

A/B Testing   You Might Be Driving in the Wrong Direction TOMASZ BORYS @Kissmetrics #KissWebinar @thuelmadsen Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before joining forces with Kissmetrics, he was a Lyft driver in SF, which is also how he ended up as a Kissmetrics marketer. Whenever Thue is not trying to automate everything around him, you can find him hiking in the Sierras. THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen Tomasz loves dipping his feet in the river while fishing, injuring his thumb while gaming, hacking away at a golf club, and driving demand at Kissmetrics. He’s also the biggest fan of gummy bears. TOMASZ BORYS Director of Marketing, Kissmetrics @tbcali @Kissmetrics #KissWebinar @tbcali 1 The optimum strategies for A/B testing 2 How we A/B Test at KISSmetrics 3 A/B Testing beyond click conversions - having a pulse on the entire funnel and why it’s important TABLE OF CONTENTS 4 The influence social traffic has in the funnel • Set a Goal • Baby Steps • Aim for Statistical Significance • Never Lose Sight THE OPTIMUM STRATEGIES FOR A/B TESTING Baby Steps People want to move the needle and see results fast, by implementing multiple changes at once. But how can you be sure what element had an impact? Aim for Statistical Signiﬁcance SET A GOAL Aim for Statistical Significance getdatadriven.com Never Lose Sight We can caught up on what we think is best to optimize conversions or traffic…but data doesn’t lie • 4,000 and above sample size • 99% Statistical Confidence • Blind Eye for 1 Week HOW WE A/B TEST AT KISSMETRICS 4,000 and Above Sample Size Significant or decent amount of data is needed   when you’re testing beyond click conversions 4,000 and Above Sample Size signiﬁcant or decent amount of data is needed when you’re testing beyond click conversions 99% Statistical Confidence Police Blind Eye for 1 Week Data can be very erratic the first several days, so it’s easy to hit the panic button A/B Testing Beyond Click Conversions ORIGINAL VARIANT SIGNUPS Signups Install JS Custom Data Custom Data Opportunities Created SOCIAL AD FUNNEL TOMASZ BORYS Director of Marketing, Kissmetrics @tbcali tborys@kissmetrics.com THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen tmadsen@kissmetrics.com Questions?

Marketing

@NEILPATEL
@Kissmetrics
#KissWebinar
@thuelmadsen

Thue is the Kissmetrics Webinar Wizard and
Marketing Ops Manager. Before joining forces with
Kissmetrics, he was a Lyft driver in SF, which is also
how he ended up as a KISSmetrics marketer.
Whenever Thue is not trying to automate everything
around him, you can find him hiking in the Sierras.
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
Tomasz loves dipping his feet in the river while
fishing, injuring his thumb while gaming, hacking
away at a golf club, and driving demand at
Kissmetrics.
He’s also the biggest fan of gummy bears.
TOMASZ BORYS
Director of Marketing, Kissmetrics
@tbcali

@NEILPATEL
@Kissmetrics
#KissWebinar
@tbcali

1 The optimum strategies for A/B testing
2 How we A/B Test at KISSmetrics
3
A/B Testing beyond click conversions - having a pulse on
the entire funnel and why it’s important
TABLE OF CONTENTS
4 The influence social traffic has in the funnel

• Set a Goal
• Baby Steps
• Aim for Statistical Significance
• Never Lose Sight
THE OPTIMUM STRATEGIES FOR A/B TESTING

Baby Steps
People want to move the
needle and see results fast, by
implementing multiple changes
at once.
But how can you be sure what
element had an impact?

Aim for Statistical Signiﬁcance
SET A GOAL
Aimfor Statistical
Significance
getdatadriven.com

Never Lose
Sight
We can caught up on
what we think is best
to optimize
conversions or
traffic…but data
doesn’t lie

• 4,000 and above sample size
• 99% Statistical Confidence
• Blind Eye for 1 Week
HOW WE A/B TEST AT KISSMETRICS

4,000 and Above Sample Size
Significant or decent amount of data is needed  
when you’re testing beyond click conversions

4,000 and Above Sample Size
signiﬁcant or decent amount of data is needed when
you’re testing beyond click conversions
99% Statistical
Confidence
Police

Blind Eye
for 1 Week
Data can be very erratic the
first several days, so it’s
easy to hit the panic button

LOG IN WITH GOOGLE
Start Your Free KISSmetrics Trial

TOMASZ BORYS
Director of Marketing, Kissmetrics
@tbcali
tborys@kissmetrics.com
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
tmadsen@kissmetrics.com
Questions?

Mais conteúdo relacionado

Mais procurados

[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven

CXL

Conversion Hotel 2016 - John Ekman

Webanalisten .nl

Featuring speakers from Optimizely Ash Alhashim, Global Director, Sales and Market Development, Optimizely Amanda Swan, Community Manager, Optimizely Adam Avramescu, Head of Customer Education, Optimizely Ever wondered how Optimizely employees use Optimizely? In this session, you’ll learn about our optimization programs, and how we use Optimizely beyond traditional A/B testing. We’ll also uncover how to optimize experiences that don’t have a traditional “conversion” goal. Our Global Director of Sales and Market Development, Ash Alhashim, will walk through an advanced experiment that takes advantage of Optimizely’s most powerful features and how he takes a unique approach to measuring success, perfect for optimizers looking to push their optimization boundaries. Amanda Swan and Adam Avramescu will also share insights from the Optiverse testing program and thoughts on interpreting experiments on properties where “conversion” is unclear.

Opticon 2015-Optimization at Optimizely

Optimizely

Featuring speakers from Rue La La, Princess Cruises Abby Lokshin, Product Manager, Mobile, Rue La La Abel Ramos, Product Owner, Shopping and Booking Web Scrum Team, Princess Cruises Testing is an iterative process, oftentimes with confusing or inconclusive results. However, every experiment has something to teach you, and all too often, critical information is left unnoticed! Learn how to look for insights hidden behind the numbers and maximize the utility of every experiment you run.

Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect...

Optimizely

Discover how you can increase the performance and velocity of your testing program. Stephen will share his experience at Conversion.com, the UK’s biggest conversion optimisation agency. You’ll learn how to develop a process for personalisation (and avoid the mistakes most people make), how to accelerate your testing (with an advanced framework for prioritisation), and how to run advanced tests on product and price.

SearchLove London 2016 | Stephen Pavlovich | Habits of Advanced Conversion Op...

Distilled

Let's make testing your new text and design ideas for your landing page as straightforward as possible. Our landing page and tracking template add the last piece to the growth marketing puzzle: A/B Testing with Google Optimize. Follow our step-by-step guide and upgrade your setup. You can show 50% of your visitors your new and hopefully better version of your landing page from here on out. Build-in reports will show you which version has a statistically significant higher conversion rate. Back up your intuition and creativity with data and build up your confidence to make the right choices. You'll be rewarded with lower cost-per-acquisitions and higher customer-lifetime-values, aka happier customers.

A/B Testing Blueprint | Pirate Skills

Pirate Skills

Most optimizers aren’t big fans of the traditional redesign cycle. They promote a continuous optimizing process. And that’s fine. But sometimes websites just suck donkey balls. And a new website is a must. Also: new companies need new websites too. So how can we stop those clients from making shitty websites? Which research methods and tools can you use in which stage of the development cycle? And how do you this?

[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...

CXL

Conversion Hotel 2016 - Dela Quist

Webanalisten .nl

Opticon 2015-Statistics in 40 minutes

Optimizely

Always Be Testing - Learn from Every A/B Test (Hiten Shah)

Future Insights

[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and Tactics

CXL

Let’s face it, getting qualified leads these days is tough. Where do you put your money? How do you distinguish the successful campaigns from the money pits?? Where do you draw the line in the sand? The answer lies in testing! Join us, as Wordstream & KISSmetrics dig into these issues. We will showcase some of the best and worst examples of marketing campaign strategies from the click- and beyond! You'll learn: - How to A/B test your PPC Ads - How to prioritize your landing page layout - Strategies for your pipeline beyond the click - How to keep the pulse of your marketing funnel for long-term success For more information on WordStream, visit www.wordstream.com.

A WordStream & KISSmetrics Webinar: A/B Testing; You’re Most Likely Doing It ...

Internet Marketing Software - WordStream

Growing Your Business Through Experimentation

Hiten Shah

Turning Data into Customers - Conversion Hotel - Peep Laja

CXL

"Conversion Optimization" and "Testing" became the hottest topics on the strategic roadmaps of online marketing managers. But still, a lot of companies suffer from easy to solve problems that probably cost a couple of million dollars. In this talk, André Morys shows some unkown conversion barriers that are not visible on the website. He shows some practical examples and cases that show how effective optimization programs gain a lot more momentum and drive more optimization ROI than usual testing efforts do.

The Million Dollar Optimization Strategy - Andre Morys - ConversionXL Live 2015

CXL

Conversion Optimization in Practice: BaconBiz 2013

Patrick McKenzie

Building Stuff To Help You Sell The Stuff You Build

Patrick McKenzie

Talks@Coursera This tech talk will describe how to build an experiment platform that can handle large-scale experiments. The talk will also discuss several best practices in designing and analyzing online experiments learned from companies like Coursera, Microsoft and LinkedIn. About the Speakers Ya Xu has been working in the domain of online A/B testing for over 4 years. She currently leads a team of engineers and data scientists building a world-class online A/B testing platform at LinkedIn. She also spearheads taking LinkedIn's A/B testing culture to the next level by evangelizing best practices and pushing for broad-based platform adoption. She holds a Ph.D. in Statistics from Stanford University. Chuong (Tom) Do currently leads a team of data engineers and analysts in the Analytics team at Coursera, which is responsible for data infrastructure and quantitative analysis in support of the product and business. He completed his Ph.D. in Computer Science at Stanford University in 2009 and worked as a scientist in the personal genetics company 23andMe until 2012, where his research has collectively spanned the fields of machine learning, computational biology, and statistical genetics.

Talks@Coursera - A/B Testing @ Internet Scale

courseratalks

Optimizely Customer Showcase - Bing Ads

Optimizely

Taming The HiPPO

Google A/NZ

Mais procurados (20)

[Elite Camp 2016] Peep Laja - Fresh Out Of the Oven

Conversion Hotel 2016 - John Ekman

Opticon 2015-Optimization at Optimizely

Opticon 2015-Hypothesize, Analyze, Optimize: Using Hidden Insights to Perfect...

SearchLove London 2016 | Stephen Pavlovich | Habits of Advanced Conversion Op...

A/B Testing Blueprint | Pirate Skills

[Elite Camp 2016] Karl Gilis - How to Make Sure Your New Website Won’t Be a F...

Conversion Hotel 2016 - Dela Quist

Opticon 2015-Statistics in 40 minutes

Always Be Testing - Learn from Every A/B Test (Hiten Shah)

[Elite Camp 2016] Phil Nottingham - CRO with Video: Tips, Tricks and Tactics

A WordStream & KISSmetrics Webinar: A/B Testing; You’re Most Likely Doing It ...

Growing Your Business Through Experimentation

Turning Data into Customers - Conversion Hotel - Peep Laja

The Million Dollar Optimization Strategy - Andre Morys - ConversionXL Live 2015

Conversion Optimization in Practice: BaconBiz 2013

Building Stuff To Help You Sell The Stuff You Build

Talks@Coursera - A/B Testing @ Internet Scale

Optimizely Customer Showcase - Bing Ads

Taming The HiPPO

Destaque

A Crash Course on A/B Testing

Stephen Edward Puiszis

The Power of A/B Testing

Alexandre Pallota

Sink or Swim? Supporting the Transition to New Manager | Webinar 08.11.15

BizLibrary

4 Steps Toward Scientific A/B Testing

Janessa Lantz

Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js

Chris Saint-Amant

Basics of AB testing in online products

Ashish Dua

User experience calendar 2011

Experience Dynamics

How to Perform A/B Testing?

QATestLab

A/B Testing Pitfalls and Lessons Learned at Spotify

Danielle Jabin

Chief Persuasion Officer Bart Schutz will be keynoting about his Wheel of Persuasion on marketing & optimization conferences throughout Europe in october & november 2014: Conversion Jam - Stockhom, Sweden Conversions@Google - Dublin, Ireland iLive - Riga, Latvia Conversion Conference - London, England Conversion Conference - Berlin, Germany Conversion Summit - Istanbul, Turkey Conversion Hotel - Texel, the Netherlands At these conferences he will present his Maxima (System 1) and Willem (System 2) story and inspire you how to persuade these two parts of your customers brains. This slideset is the main slideset for each conference and is extended with extra A/B-testcases. More info: http://wheel-of-persuasion.com (the knowledge source) http://online-dialogue.com (hire our team) http://testing.agency (fast and quality A/B-test development) http://abtestguide.com (you A/B-test workflow tool) and http://Persuasion.tips for an overview of Bart his recent work.

Keynote Wheel of Persuasion - Bart Schutz of Online Dialogue on how to persua...

Online Dialogue

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing ...

Online Dialogue

Experimentation Platform at Netflix

Steve Urban

City of You - Branding of the City of Youngstown

DOYO Live

The smart-marketers-guide-to-webinar-marketing

Daniel Howard

See the video replay here: http://www.marketingsherpa.com/video/webinar/adobe-conversion-lift-webinar-strategy Daniel Burstein, Director of Editorial Content, MECLABS, interviews Shelby Britton, Senior Product Marketing Manager, Adobe, on how it was able to revamp its webinar strategy for a huge lift in conversions, placing webinars as its second largest generator of sales. Watch as Daniel and Shelby answer your questions about using webinars to drive conversion.

Webinar Marketing: How Adobe saw a 500% lift in conversion from changing its ...

MarketingSherpa

User experience is a huge buzzword in the design world right now, but what does it really mean? The truth is it means lots of things, and can best be thought of as a philosophy for creating engaging experiences for digital points-of-contact. In my talk, I’ll introduce you to tools, best practices, and approaches to design that leverage user goals and needs to build better products of all stripes. Guiseppe Getto, Ph.D. is a college professor based in North Carolina and is President and Co-Founder of Content Garden, Inc., a digital marketing and UX consulting firm. He consults with a broad range of organizations who want to develop better customer experiences, better writing, better content, better SEO, better designs, and better reach for their target audience. He has taught at the college level for over ten years. During that time, he has also consulted and formed service-learning partnerships with many non-profits and businesses, from technical writing firms to homeless shelters to startups.

Transforming Customer and Client Outcomes Through Engaging User Experiences

DOYO Live

Webinar marketing sucess

Faraad Coston

We are very excited to announce that PALO Creative is the title agency sponsor for DOYO Live and that Jim Komara, Digital Marketing Director at PALO will be a breakout session leader and will give you a deep dive into the world of SEO in a fun understandable way. The Google algorithm has become very sophisticated and many traditional SEO techniques no longer work. Black Hatters and Link Farmers are largely out of work with Content Producers coming into their own. This session will review some of the recent changes that affect your search engine rank and dive into the details that make the difference. After this session, attendees will be able to “see like a search engine spider.” About Jim Komara Jim began launching websites in the late 1990s when the primary way to find a site was using directories like Yahoo. Shortly after starting his first web design business, Google came on the scene and suddenly customers were all vying for the number one spot on search engine results. Jim started writing computer games and math programs in high school. A veteran of the United States Army, he served in Operation Desert Storm during the Gulf War as an Intelligence Analyst, building computer databases. After returning home, he worked as a real estate agent and began writing programs for his business and creating websites. In 2004, founded RealEstatePointe, a company that creates and maintains real estate company websites and web applications across the country. Since then he has focused on digital marketing, SEO and website building for companies across a broad spectrum of industries. Jim is the Digital Marketing Director at PALO Creative where he gets to meet new clients who bring new challenges every week, which he calls “non-stop fun!”

See Like a Spider

DOYO Live

In today’s world, we each carry the world’s knowledge in our pants. As consumers, our uninhibited ability to access self-service information through our smart phones and other mobile devices has forever changed how and when we research, consider and ultimately buy products and services. Research from the Marketing Leadership Council even goes as far as to suggest that approximately 57% of a purchase decisions are now made prior to the buyer ever communicating with a sales representative. Sign up today for DOYO Live! In this high-energy presentation, Nate Riggs will open your mind to the philosophies, methodologies and technologies behind Inbound Marketing – a highly progressive form of marketing that meets this change in consumer behavior head on. Nate’s presentation has been designed to help you shift your mindset, while empowering you to begin the journey of using the inbound marketing approach to capture customer attention, build trust through information sharing and turn website visitors into leads and customers. Audience members will come away from this presentation with a better understanding of: How the purchase funnel has changed as a result of digital media. The tie between online content and lead generation, and how you can increase demand for your products and services. How marketing automation and personalization can increase sales. Where social media fits in your company’s your marketing mix. How to plan your internal resources to execute inbound marketing strategies over the long term. Nate Riggs was one of the featured keynote speakers at DOYO Live – A digital marketing + interactive design conference held in Youngstown, Ohio. DOYO Live 2017 will take place on August 3, 2017.

How to Get Sharper Results from Marketing & Sales

DOYO Live

Destaque (19)

A Crash Course on A/B Testing

The Power of A/B Testing

Sink or Swim? Supporting the Transition to New Manager | Webinar 08.11.15

4 Steps Toward Scientific A/B Testing

Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js

Basics of AB testing in online products

User experience calendar 2011

How to Perform A/B Testing?

A/B Testing Pitfalls and Lessons Learned at Spotify

Keynote Wheel of Persuasion - Bart Schutz of Online Dialogue on how to persua...

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing ...

Experimentation Platform at Netflix

City of You - Branding of the City of Youngstown

The smart-marketers-guide-to-webinar-marketing

Webinar Marketing: How Adobe saw a 500% lift in conversion from changing its ...

Transforming Customer and Client Outcomes Through Engaging User Experiences

Webinar marketing sucess

See Like a Spider

How to Get Sharper Results from Marketing & Sales

Semelhante a A/B Testing: You Might be Driving in the Wrong Direction

Beyond Click Conversions – KISSmetrics’ Director of Marketing Tomasz will cover why it’s so crucial to have a pulse on the entire funnel when A/B Testing. Tomasz will be showing an actual test he ran for this specific demo, showcasing the results from trial conversions, down to those signups that have interacted with the app, to those leads that converted. What you’ll learn: - The influence social traffic has in the funnel - The optimum strategies for A/B testing - How we A/B Test at KISSmetrics - A/B Testing beyond click conversions – having a pulse on the entire funnel and why’s its important

A/B Testing: You Might Just Be Doing it Wrong

Sprout Social

Ask any CEO what they want from their marketing team and they'll all say the same thing: growth. But with so many different things you could do, how do you know what you should do to unlock new growth? The answers you're looking for are hiding in your data. In this presentation we'll cover exactly how you can analyze your data to uncover new growth opportunities and achieve your goals. You'll learn how to: Slice your data to find new growth opportunities Validate ideas with simple tests before committing to expensive initiatives Use data to drive product changes that maximize growth You'll see real-world examples of how the growth teams at Airbnb, GrowthHackers and Facebook use this same process to unlock new growth from smart data insights.

How to Find the Big Growth Opportunities Hiding in Your Data

A/B Testing: You Might be Driving in the Wrong Direction

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (19)

Semelhante a A/B Testing: You Might be Driving in the Wrong Direction

Semelhante a A/B Testing: You Might be Driving in the Wrong Direction (20)

Mais de Kissmetrics on SlideShare

Mais de Kissmetrics on SlideShare (20)

Último

Último (20)

A/B Testing: You Might be Driving in the Wrong Direction