The document discusses A/B testing techniques for validating new product launches. It begins by defining A/B testing as showing two or more versions of a product to different random user groups to determine a winning variant. For new products, the document recommends first gathering qualitative feedback through methods like focus groups before establishing a minimum viable product (MVP) baseline and using A/B tests with non-inferiority metrics. It then suggests iteratively testing other features against the MVP through A/B tests and establishing a holdout group for long-term comparison to determine the winning variant ready for full launch. Other validation methods like focus groups, support staff feedback, and limited beta testing are also outlined.

1. www.productschool.com
A/B Testing for New Product Launches
by Booking.com Sr PM

2. CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™
20 HOURS40 HOURS40 HOURS

3. Corporate
Training
Level up your team’s Product
Management skills

4. Free Product Management Resources
BOOKS
EVENTS
JOB PORTAL
COMMUNITIES
bit.ly/product_resources
COURSES

6. A/B testing new products
BY : Namrata Panwar
Sr. Product Manager, Booking.com [Dec 2020]
LinkedIn : https://www.linkedin.com/in/namratapanwar/

7. About me
● Namrata Panwar, Sr PM @ Booking.com, Amsterdam
● 8 years of experience
● Current : Working on partner facing products - onboarding,
performance, pricing for Home (Airbnb - like) partners
● Previously : with customer facing products in - E-commerce
startups
● Started career with Reliance Industries, as PM for Jio Mobile Apps
● Hobbies : Travelling, adventure sports, decorating my new house

8. Existing vs New product development cycle
A/B testing : What and Why?
01
02
Qualitative data vs quantitative data
03
Steps to NPD testing via A/B test
04
Recap
Agenda

9. ● Know basics about A/B testing
● Overview of a product development lifecycle
● Know about A/B testing techniques for new products
● Other ways for validating new products- apart from A/B testing
By the end of the
presentation....

10. A/B testing : What?
Base version Another version
In this example : Adding a buffer icon
before loading search results of a flight in
order to increase page views
Objective : To choose winning variant
based on a hypothesis
Showing 2 (or more ) versions to different
set of similar users, chosen randomly
Control A Variant B
V/S
Variant B Winning variant B!
50% 50%

11. Examples
Different landing cards allows the platform to tap into the psychology
of its viewers
Asking repeat users to ‘Continue’ with registration lead to more users
completing it

12. Making low risk modifications
(that can be easily undone)
Understanding users and solving user
painpoints
A/B testing : Why?
Introducing features and feature changes

13. Typical Product development cycle
Ideate
Design &
develop
Launch and
Iterate
Maturity
01
02
03
04
Lesser ambiguity, More experience

14. Existing product development
For such products, iterations are mostly
trials- regular A/B or multivariate testing
Fail fast : optimizing the user experience
to improve an existing product
Existing product : A product base exists
with reasonable user adoption
Tiny optimizations can have a big
impact -example changing the way
Microsoft’s search engine displayed ad
headlines increased revenue by an
astonishing 12%
Approach 1:

15. New product development
Example : Polaroids or Google extending
its product lines such as Google flights or
Google Health
Most amount of time is spent between
ideation and launch, focus on product
definition and qualitative feedback
New product : Establishing a new base
product from scratch -solving new
problems
No base for traditional A/B
testing
Approach 2:

16. No prior definition of success
Limited set of users/ testers
No base to iterate on
Traditional A/B testing :
Challenges with new
products

17. NPD testing framework

18. NPD testing
Step 1 : Qualitative data over quantitative data
Start with extensive qualitative research
Invest in getting design and research right upfront
Use Qualitative metrics that validates your product
: e.g. User satisfaction, Upvotes, Likes etc
Objective : Maintain high level of product quality
BEFORE you start A/ B testing and lower risk
Design
and build
Get Qualitative
feedback

19. NPD testing
Step 2 : Establish foundations
Launch this MVP to create a foundation-
using A/B tests with Non inferiority
success metrics
Decide on your MVP - Minimum Version of your
Product
MVP BASE
MVPFINAL PRODUCT
W/o hurting metrics

20. NPD testing
Step 3 : Iterate, Measure success
Short term success: Running A/B tests to
introduce other modules -decide winning
variants
Long term success : Setting up a holdout
: excluding a small % of your users from
all tests for a long time (typically 6 mo- 1
yr) to compare performance
BASE
A/B test
V1
V2
10%
90%

21. NPD testing
Outcome : Winning variant ready to be ‘launched’ in market
V1 FINAL PRODUCT
Ready for launch
Winning variant
=

22. Some other ways of validating new products
FOCUS GROUPS
Invite a group of similar users and
giving them a trial/ demo
SUPPORT STAFF
Validate the product by those that
interact most with your users :
customer support, local offices, etc
BETA testing
Launching your product to a limited
set of users and getting extensive
feedback, before release to all public
High effort/ $$$Low effort/ $

23. Depending on new vs existing product you will focus on
different lifecycle phases
A/B testing : Effective and easy way to learn and remove
biases 01
02
For NPD, define your MVP and introduce your base
(if it passes Non inferiority condition) 03
Measure long term performance of your new
product via holdouts 04
Recap
05Focus groups and Beta testing are other ways of
validating new products

24. CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon, and infographics & images by Freepik
THANKS!
Do you have any feedback or questions?
Email : namrata16.it@gmail.com
Sr. Product Manager, Booking.com
LinkedIn : https://www.linkedin.com/in/namratapanwar/

25. www.productschool.com
Part-time Product Management Training Courses
and
Corporate Training