Will 2020 mark the shift to a remote-first world in the long run? For GitHub, a distributed workforce is nothing new. Join Sha Ma, VP of Engineering, and Gregory Ceccarelli, Director of Data Science, to learn how they built and scaled a successful experimentation program. They'll share their experience implementing Optimizely across timezones, a remote workforce, and a new business model. In this session, you'll learn how to: Optimize UX for a freemium business model Use data to deliver customer-centered products Scale experimentation and accelerate growth

1. Shipping to Learn and
Accelerate Growth
Opticon 2020
Presented by @shama0 & @gregce

2. Greg Ceccarelli (@gregce)
Director, Data Science
Data & Security Product
Sha Ma (@shama0)
VP, Software Engineering
Core Platform & Ecosystem

3. Our Story
GitHub started in 2008 as a way
to make it easier for developers
to host and share their code.

4. 50M+
Our global community today
87M+
Developers globally building on GitHub in
2020; we want to hit 100M developers by
2025.
Of users are contributing from outside of
the US.
Pull requests merged in 2019 - and 28%
more developers opened their first pull
request in 2019 than in 2018.
80%
100M+Repositories across every continent
on earth.

5. 2K
Rapid internal growth
15+
Employees at GitHub, significant growth
in the last year.
Employees work outside of our San
Francisco HQ, distributed across the globe.
Countries and regions, where full time
employees live and work. We hire in all 50
States of the US.
70%

6. Problem Statement
Product Development at this
scale can be costly

7. 1. User Experience Research
2. Focus on Minimum Viable Product
3. Data Driven Experimentation
Ship to learn in practice

8. UX Research:
The story
behind GitHub
Actions
All customers found the
Actions sidebar
valuable
Customers thought the
suggestions were very
helpful for figuring out
how to edit the config
file
Customers wanted to
see what ‘Variables’
included - secrets?
“Want to see some mechanism
(global variables or something
else) that would allow me to
retrieve that value from within
the build. And some way to say
only I can see this....” -
Customer X

9. MVP:
Starting small
with Discussions

10. Vision:
Home for All
Developers
“We want every developer and team
on earth to be able to use GitHub for
their development, whether it’s
private or public development.”

11. We now have to learn what to ship
● How to understand new behavior as developers start to
adopt a fundamentally different product
● How to reorient our Revenue function’s strategy to grow
our new freemium self serve business
● How to apply developed product pricing principles to
package our SKUs

12. Meanwhile...
We’re amidst a
global pandemic

13. Developer
Signups
GitHub Free’s launch premise
Free Orgs Creating
Private Repos
Increase the YoY growth rate in the volume
of new signups
Increase the YoY growth rate in the count of
developers that consume content on GitHub
Monthly
Contributors
Remove barriers to entry for small teams to
use the core GitHub workflow and
accelerate private repo adoption
Monthly
Engaged Users
Increase the YoY growth rate in the count of
developers that create content on GitHub

14. And in early 2020...
Implemented Optimizely to
replace our homegrown platform

15. But… experimentation is key to
learning what matters most to our
customers

16. Why Experiment?
Experimentation allows us to
measure the quantitative causal
impact of what we’re building

17. Causation
allows us to
isolate the
impact on Y
because of X It is often easier to generate a plan,
execute against it, and declare success,
with the key metric being: "percent of plan
delivered," ignoring whether the feature
has any positive impact to key metrics
(Kohvai et al. 2013)

18. Experimentation at
GitHub is enabled
via people, process
and technology
Hubbers Propose &
Design Experiments

19. Our Experimentation
Council
● Ensures cross
functional partnering
across our org
● Accelerates our ability
to communicate
impact against our Key
Metrics
GitHub
Product FinanceMarketing
Experimentation
Council

20. Putting this all together...

21. At GitHub, MVPs come in two flavors

22. 1. A Hubber logs an experiment
proposal issue in the Experiment
Council repo
2. The team meets weekly on Wednesdays
to review and approve proposals
3. Next a Data Scientist is assigned
to issues determine metric
baselines and power (if required)
4. The experiment is then instrumented
in Optimizely by Engineering
5. Results are measured and
conclusions about the hypothesis
are drawn
1. Proposer (e.g. Product) makes a
decision about whether to ship the
feature or go back to the drawing
board
Focusing in on how
we actually
experiment...

23. Process out of the way...
Now let’s chat about some
experiments!

24. 🧪 Can we encourage repo growth within Orgs?
Hypothesis: “If we make our Org UI more informative, then more users will
create repos in their Org.”
Control Variant

25. 🧪 Can we encourage more Org page activity 🎓
Hypothesis: “If we make org pages easier to find, then viewership and org
activity will commensurately increase.”
Control Variant

26. Our proposal pipeline has accelerated

27. 1. Pipeline Inspiration: # of proposals in
pipeline
1. Process Velocity: % of proposals run and
the average latency to go from proposal to
approved experiment to instrumented and
shipped experiment
1. Quality: # of shipped experiments without
issue (customer facing or related to
measurement) and # of experiments shut
down or shipped as features
Current
Program
Metrics 📈

28. Experimentation in their words
Carmel Schetrit (@Carmel-S)
Demand Generation Manager
Growth
“The experimentation council
helped the Growth team in 2020
to evaluate the complexities of
designing, instrumenting, and
measuring experiments on
github.com with Optimizely.
The council often shed light on
scenarios requiring more careful
consideration, including those
that conflict with concurrent
experiments or cannot reach
statistical significance.”

29. Experimentation in their words
Katie Sipos (@ohitsmekatie)
Senior Product Manager
Education
“Our team is rolling out changes to
our GitHub Classroom onboarding
flow through Optimizely.
Being able to definitively prove
which experience is best for our
users and then ship that with
confidence is great!
Without it, we would be flying blind
and shipping features with our gut
feelings and not data.”

30. Now that’s a wrap!
Closing thoughts

31. Accelerating our process
1. Top of Funnel: Have a healthy
proposal pipeline and backlog
1. Middle: Diagnose and triage process
bottlenecks (e.g. do you have
enough designers?)
1. Bottom: Invest in integrating
downstream product metric
enrollment to take full advantage of
Optimizely’s Stats Engine

32. Accelerating our program maturity
1. Strategic Prioritization: # of
experiments selected based on priority
score (versus last in, first out) and
alignment with OKRs
1. Culture: # of times we broadly
syndicate learnings and % business
functions integrated into the process @
GitHub