O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Experimenting on Humans

518 visualizações

Publicada em

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1zNFo1X.

Aviran Mordo and Talya Gendler explain how their engineering team is supporting product managers in making the right decisions and getting their product roadmap on the right path. Filmed at qconsf.com.

Aviran Mordo is the head of back-end engineering at Wix. Talya Gendler is Senior Software Engineer at Wix.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Experimenting on Humans

  1. 1. Experimenting on Humans Aviran Mordo Head of Back-end Engineering @aviranm http://www.linkedin.com/in/aviran http://www.aviransplace.com Talya Gendler Back-end Team Leader www.linkedin.com/in/talyagendler
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /experimenting-humans
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  4. 4. Wix In Numbers Over 55M users + 1M new users/month Static storageis >1.5Pb of data 3 data centers + 3 clouds(Google,Amazon,Azure) 1.5B HTTP requests/day 800 peoplework at Wix, of which ~ 300 in R&D
  5. 5. Basic A/B testing Experiment driven development PETRI – Wix’s 3rd generation open source experiment system Challenges and best practices Complexities and effect on product Agenda
  6. 6. 15:40 A/B Test
  7. 7. Home page results (How many registered)
  8. 8. This is the Wix editor
  9. 9. Our gallery manager What can we improve?
  10. 10. Is this better?
  11. 11. Product Experiments Toggles & Reporting Infrastructure
  12. 12. How do you know what is running?
  13. 13. If I “know” it is better, do I really need to test it? Why so many?
  14. 14. Sign-up Choose Template Edit site Publish Premium The theory
  15. 15. EVERY new feature is A/B tested We open the new feature to a % of users  Measure success  If it is better, we keep it  If worse, we check why and improve If flawed, the impact is just for % of our users Conclusion
  16. 16. Start with 50% / 50% ?
  17. 17. New code can have bugs Conversion can drop Usage can drop Unexpected cross test dependencies Sh*t happens (Test could fail)
  18. 18. Language GEO Browser User-agent OS Minimize affected users (in case of failure) Gradual exposure (percentage of…) Company employees User roles Any other criteria you have (extendable) All users
  19. 19. First time visitors = Never visited wix.com New registered users = Untainted users Not all users are equal
  20. 20. Start new experiment (limited population)
  21. 21. Adding a mobile view
  22. 22. First trial failed Performance had to be improved
  23. 23. Halting the test results in loss of data. What can we do about it?
  24. 24. Solution – Pause the experiment! • Maintain NEW experience for already exposed users • No additional users will be exposed to the NEW feature
  25. 25. PETRI’s pause implementation Use cookies to persist assignment If user changes browser assignment is unknown Server side persistence solves this You pay in performance & scalability
  26. 26. Decision Keep feature Drop feature Improve code & resume experiment Keep backwards compatibility for exposed users forever? Migrate users to another equivalent feature Drop it all together (users lose data/work)
  27. 27. Numbers look good but sample size is small We need more data! Expand Reaching statistical significance 25% 50% 75% 100% 75% 50% 25% 0%Control Group (A) Test Group (B)
  28. 28. Keep user experience consistent Control Group (A) Test Group (B)
  29. 29. Signed-in user (Editor)  Test group assignment is determined by the user ID  Guarantee toss persistency across browsers Anonymous user (Home page)  Test group assignment is randomly determined  Can not guarantee persistent experience if changing browser 11% of Wix users use more than one desktop browser Keeping persistent UX
  30. 30. Always exclude robots Don’t let Google index a losing page Don’t let bots affect statistics
  31. 31. # of active experiment Possible # of states 10 1024 20 1,048,576 30 1,073,741,824 Possible states >= 2^(# experiments) Wix has ~200 active experiments = 1.606938e+60
  32. 32. Supporting 2^N different users is challenging How do you know which experiment causes errors? Managing an ever changing production env.
  33. 33. Override options (URL parameters, cookies, headers…) Near real time user BI tools Specialized tools
  34. 34. Integrated into the product
  35. 35. Why should product care about the system architecture
  36. 36. Share document with other users
  37. 37. Document owner is part of a test that enables a new video component
  38. 38. What will the other user experience when editing a shared document ? Owner Friend
  39. 39. Assignment may be different than owner’s Owner (B) Friend (A)
  40. 40. Enable features by existing content Enable features by document owner’s assignment Exclude experimental features from shared documents Possible solutions
  41. 41. Petri is more than just an A/B test framework Feature toggle A/B Test Personalization Internal testing Continuous deployment Jira integration Experiments Dynamic configuration QA Automated testing
  42. 42. Petri is now an open source project https://github.com/wix/petri
  43. 43. Q&A Aviran Mordo Head of Back-end Engineering @aviranm http://www.linkedin.com/in/aviran http://www.aviransplace.com Talya Gendler Back-end Team Leader www.linkedin.com/in/talyagendler https://github.com/wix/petri http://goo.gl/L7pHnd
  44. 44. Credits http://upload.wikimedia.org/wikipedia/commons/b/b2/Fiber_optics_testing.jpg http://goo.gl/nEiepT https://www.flickr.com/photos/ilo_oli/2421536836 https://www.flickr.com/photos/dexxus/5791228117 http://goo.gl/SdeJ0o https://www.flickr.com/photos/112923805@N05/15005456062 https://www.flickr.com/photos/wiertz/8537791164 https://www.flickr.com/photos/laenulfean/5943132296 https://www.flickr.com/photos/torek/3470257377 https://www.flickr.com/photos/i5design/5393934753 https://www.flickr.com/photos/argonavigo/5320119828
  45. 45. Modeled experiment lifecycle Open source (developed using TDD from day 1) Running at scale on production No deployment necessary Both back-end and front-end experiment Flexible architecture Why Petri
  46. 46. PERTI Server Your app Laboratory DB Logs
  47. 47. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/ experimenting-humans