Wojciech Galuba, Decision Tools Lead, Facebook
Experimentation is a valuable tool for supporting product decisions, iterating on features and gaining actionable insights into people's behavior.
In this session Wojciech Galbua, Data Scientist at Facebook, presents an overview of Facebook's experimentation framework and how it is used to make day-to-day data-driven decisions at global scale.
The talk focuses on the challenges of building and scaling the analytics infrastructure, designing the tools for ease-of-use and ensuring broad adoption of sound experimentation methodologies across all the teams.
4. History of Data Science Infra at FB
• Founded April 2012
• A group of data scientists and software engineers
• Experienced first hand the need for better infrastructure
• Need continues to grow
• Team doubled over the past year
• Expect continued rapid growth this year
13. Many metric dimensions
• Different contexts of user actions
• Thousands of device types
• Geography
• Demographics
• Time
• Enormous space of possible questions
14. Many teams
• Many ways to run an experiment
• Diverse audience for results
• Huge set of results from every experiment
• Many ways to interpret results
24. Computing answers to exponential
number of possible questions
Pre-compute
• low specificity
• low dimensionality
• long-term
Compute on-the-
fly
• high specificity
• high dimensionality
• short-term
A balancing act
37. Understanding uncertainty
“Robust misinterpretation of confidence intervals”
Rink Hoekstra et al.
Psychonomic Bulletin & Review
• Only 3% of scientists got
all 6 answers right...
• How do we educate the
users of the tools?