1.
Building an Experiment
Pipeline for GitHub’s New
Free Team Offering
Philip Bremer
Director, Software Engineering
GITHUB

2.
Building an experiment
pipeline for GitHub’s
Free for Teams plan
GitHub
Presented by Philip Bremer

3.
● Director of Software Engineering
UI Platform
● Korean adoptee immigrant
● Soccer, yoga, skiing
● Current book: The New Jim Crow
● Ambivert: 75% introvert, 25% extrovert
● Pronouns: he/him
About me
@philip_bremer

4.
GitHub’s growing product footprint

5.
● Individuals and teams
● Price should not be a barrier
● Free private repos
● Upgrade path for advanced
features
Free for Teams

6.
Time to build an
experiment
pipeline!

7.
New orgs and teams
Global Nav Repo CTAsOrg Creation

8.
Global nav experiment
Problem: Users can’t easily find their organization(s)
after creating or joining
Hypothesis: Adding a link to “Your organizations” in
user’s global dropdown menu will improve discovery
Results:
● Success!
● +134.46% improvement
● Oops! We spent a ton of impressions
● ~10% of total impression budget

9.
● Misunderstood the difference
between bucketing vs
activation vs tracking
● High traffic for global nav
● The experiment baked too long
How did we spend
so many
impressions?

10.
Track
Log notable events
to calculate impact
of variation
Bucketing, activation, and tracking
Bucketing
Assign user
to variation
Activation
Log impression
when user sees
the experiment

11.
● Anonymous sessions are
read-heavy and repetitive
● Cache anonymous sessions to
reduce load on our application
● Goal: serve page for a subset of
anonymous sessions without
disrupting the benefits of our cache
● Goal: serve page with no negative
impact to our page load budget
Logged out experiments

12.
Performance & DX
● Robust datafile management
● More internal API guardrails for
activate vs get_variation
● Improved logging and testing
across environments
Experimentation culture
● Empower feature teams to
experiment with evidence-
based delivery
Up Next?

13.
Thank you!