1. Controlled Experimentation to
Guide Product Innovation!
!
Rajesh Parekh!

2. Controlled Experimentation (A/B Testing)!
• Method to study effects of a treatment
#
• Concept:!
- Randomly split users into two groups#
Randomly)
Divide) A : Control#
B: Treatment#
- A and B are identical to each other except
A)(Control)) B)(Treatment)) for the treatment being evaluated#
- Collect performance metrics from the
experiment#
- Run statistical tests to determine if
differences between A and B are purely
by chance#
Measure)&)
Evaluate)
Controlled)Experimenta=on)Panel) 2

3. Why Run Controlled Experiments?!
• Commonly used approach in clinical trials!
- What is the effect of a particular drug / treatment?#
• Systematically validate hypotheses with data!
!
• Concurrently run the treatment and control!
- The difference (if any) is#
Because of the treatment OR#
Due to random chance#
• Determine if a treatment is causal in nature!
- E.g., Making the search box bigger causes increase in queries / user#
Controlled)Experimenta=on)Panel) 3

4. Controlled Experimentation: Use Cases!
#
A"B"Stract"Widget"Company" A"B"Stract"Widget"Company"
_________________) _________________)
_________________) _________________)
_________________) _________________)
_________________) BUY)NOW) _________________) BUY"NOW"
Website)Variants)
Controlled)Experimenta=on)Panel) 4

5. Controlled Experimentation: Use Cases!
#
Free)Trial) Play)Now)
Mobile)Call)to)Ac=on)
Controlled)Experimenta=on)Panel) 5

6. Controlled Experimentation: Use Cases!
#
Top)deal)highlighted)
Email)Template)Design)
Controlled)Experimenta=on)Panel) 6

7. Controlled Experimentation: Use Cases!
#
Backend)changes)(e.g.,)Personaliza=on)Algorithm))
Controlled)Experimenta=on)Panel) 7

8. Controlled Experimentation: Use Cases!
# • Follow-up message for users
who previously clicked on an
ad#
• Incentive campaign to re-
engage lapsed users#
• Think of this as placing filters /
guards on a randomly chosen
user population#
Custom)Defined)User)Segments)
Controlled)Experimenta=on)Panel) 8

9. Key Components of an Experimentation Platform!
Hashing function! Metrics – suite of KPI!
Group)0)
! ! Revenue"
!
!
F()))))))))))))) Group)1) !
!
Time"Spent"
Abandonment"
Click>Through"Rate"
! Group)NU1) !
! ! Session"Length" Purchase"Rate"
Logging! Dashboard!
!
!
!
!
• Detailed)logging)of)all)user)interac=ons) • Metric)improvements)and)Sta=s=cal)
Significance)in)a)central)place)
Controlled)Experimenta=on)Panel) 9

10. Ensure Identical Control and Treatment!
Gender""
• Custom Segments#
Male)
Female)
• Frequency Distribution#
CONTROL" TREAMENT"
Region"Size"
Small)
Medium)
Large)
CONROL" TREATMENT"
Prior"Exposure"
• Large Difference in Prior
Exposure Rate violates δ%"
assumptions# No)
Yes)
CONROL" TREATMENT"
Controlled)Experimenta=on)Panel) 10

11. A/A Tests!
• Run an experiment with two identical variants#
• Helps to determine if:#
- Users are being split uniformly at random#
- Correct data is being logged#
- Variance between identical populations of users is acceptable#
• Challenge:!
- Few purchases of high value deals render statistically significant
difference between treatment and control#
SPAIN"TRIP"
$1,999"
Controlled)Experimenta=on)Panel) 11

12. Monitor Each Variant!
• Place yourself in each variant
to validate the experience#
!
• Wrong sort order!!
!
!
Carefully)inspect)each)variant)
Controlled)Experimenta=on)Panel) 12

13. Objective Function!
#
#
Conversion" Revenue"
P(conversion)) E(rev))=)P(conversion))*)price)
) )
• Favors)lower) • More)expensive)deals)can)
price)deals) dominate)
Need)to)balance)mul=ple,)oZen)conflic=ng)objec=ves))
Controlled)Experimenta=on)Panel) 13

14. Measure Overall Impact!
• Test focuses on#
- A particular area of
the website#
- A sub-population of
users#
• Measure!
- Improvement on the
sub-segment AND#
- Entire population!#
Measure)overall)impact)to)guard)against)cannibaliza=on)
Controlled)Experimenta=on)Panel) 14

15. Panel Discussion: Questions!
Controlled)Experimenta=on)Panel) 15

16. Acknowledgements#
Thanks to many talented individuals at Groupon I am privileged to work with!#
• Data Science#
• Engineering#
• Marketing / Market Research#
Controlled)Experimenta=on)Panel) 16

17. Rajesh Parekh!
Groupon!
rajesh@groupon.com!
!
Controlled)Experimenta=on)Panel) 17