It Worked for Ustream
Product Development @ Ustream
Making decisions by assumptions:
“Let’s do this, because it worked for Spotify!”
- you will never know why it worked for them.
Making decisions by experiments:
“Let’s try this out, measure eﬀectiveness,
and see if it works!
What to test?
Icon & text: +12% conversion.
Myth #13: Icons enhance usability
What to test?
• CTA microcopies: “Get started”, “Try out for free”, “Sign up”, “Learn more”
• Layout: arrangement of form ﬁelds, mixing elements on the whole page.
• “Meta parameters” of a product: trial period length, price.
How to measure effectiveness?
Conversion (basically any kind of your metrics):
• Increase in purchases,
• Number of clicks,
• Number of signups,
• Form completions,
Always question assumptions
but be prepared that only 1 out of 8 A/B tests
will drive you to signiﬁcant changes.*
*based on the tests done by appsumo.com
How Obama raised $60M by running a simple experiment
Optimizely, Dan Siroker
Who to blame for the slow loading up? Facebook or iOS?
How to set up a test?
• Targeting (when the test should start, visitor characteristics &
segmentation, technical circumstances etc.),
• Experiment type (A/B, multivariate, multi page),
• Trafﬁc allocation,
• Integration with other kind of analytics.
Technical background of a test
• Quick and dirty methods
• Always having a usable fallback is anything goes wrong
• Backend for bigger and more complicated tests
• GA and Optimizely events are our best friends
Understanding your results
Chance to beat baseline.
…or you can simply use isvalid.org anytime.
Changing directions: SaaS vs. content.
Started with concluding tiny preliminary experiments: less content,
pro CTAs. Ended up with split testing the whole page.
Ustream homepage test results
• Tested on approx. 700k unique visitors.
• No important general metrics harmed (pages / visit time on site & time on
• 1% increase in purchases starting from the home page.
• Showing 1/3 of content resulted only in halving the clicks.
A DIFFERENT APPROACH FOR
Ghetto test: Pay with PayPal
• Built by frontend engineers, only triggered from Optimizely
• Decided not to build it because of low conversion rate (~2%)
Pro Purchase Flow: 1 or 2 steps?
• Built by frontend engineers, only
triggered from Optimizely
• Currently live, analysis is on the way
Parece que tem um bloqueador de anúncios ativo. Ao listar o SlideShare no seu bloqueador de anúncios, está a apoiar a nossa comunidade de criadores de conteúdo.
Atualizámos a nossa política de privacidade.
Atualizámos a nossa política de privacidade de modo a estarmos em conformidade com os regulamentos de privacidade em constante mutação a nível mundial e para lhe fornecer uma visão sobre as formas limitadas de utilização dos seus dados.
Pode ler os detalhes abaixo. Ao aceitar, está a concordar com a política de privacidade atualizada.