O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Selenium at
Salesforce Scale
David Louvton
Lead Engineer
dlouvton@salesforce.com
Sagar Wanaselja
Lead Engineer
• About Salesforce engineering
• Operating a selenium farm
• Selenium design patterns
• Overcoming test failures
• Q&A
Age...
Salesforce Scale
• 3:2 ratio for developers : quality engineers
• everyone is a software engineer; role varies
• selenium ...
Why Salesforce loves WebDriver
WebDriver is the only test framework in our current
arsenal that truly mimics an end-user’s...
UI Testing Portfolio
• 50,000 unique Selenium tests
• 7,500 unique WebDriver tests
• full “browser compatibility”: IE6-IE1...
1,000,000 browser tests per day
• Click here to add bullets
The Selenium Infrastructure
● Automatically creating, assigning, closing and reopening
bug reports for test failures and f...
Issues of scale
• Assigning test failures to the responsible engineer
• Non deterministic tests (flappers) and re-runs
• S...
Salesforce UI Testing Pattern
• Click here to add bullets
Proper Page Objects Encapsulate Selenium
• Make an API of your page for your test
• No imports for “openqa” or “selenium.o...
Challenges for Mobile Platforms
• Adapting page objects to platform specific UI elements
• Screen sizes and adaptive layou...
Sagar Wanaselja
Lead Engineer
swanaselja@salesforce.com
Problematic WebDriver Test
Better WebDriver Test
PageObject WebDriver Test
Diagnosing Test Failures
• Screenshot and HTML capture at failure point
• AssertionError or RuntimeException
• Differences...
Pro Tips
• Version tests and app code (and configuration) together
• Use the Page Object model
• Use the API to set up the...
More Pro Tips
• Avoid using xpath selectors
• Complex DOM makes xpath evaluation much slower
• Some xpaths are not properl...
Mature Quality Strategies
• Selenium for dynamic security vulnerability analysis
• Pixel perfect image diffs
• Accessibili...
selenium meetup sf talk march 2014 Selenium at Scale
selenium meetup sf talk march 2014 Selenium at Scale
Próximos SlideShares
Carregando em…5
×

selenium meetup sf talk march 2014 Selenium at Scale

1.683 visualizações

Publicada em

a talk by David Louvton and Sagar Wanaselja about the Selenium Infrastructure at Salesforce.com and design patterns for test automation.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

selenium meetup sf talk march 2014 Selenium at Scale

  1. 1. Selenium at Salesforce Scale David Louvton Lead Engineer dlouvton@salesforce.com Sagar Wanaselja Lead Engineer
  2. 2. • About Salesforce engineering • Operating a selenium farm • Selenium design patterns • Overcoming test failures • Q&A Agenda
  3. 3. Salesforce Scale • 3:2 ratio for developers : quality engineers • everyone is a software engineer; role varies • selenium committers on staff (Jim Evans, Luke Inman) • Data centers with thousands of VMs for automation • multimillion dollar investment • using vmware (VShere), OpenStack and EC2 • 500 commits per day • ~120 software engineering teams
  4. 4. Why Salesforce loves WebDriver WebDriver is the only test framework in our current arsenal that truly mimics an end-user’s experience • We don’t mock anything • We test against a live app • We suffer the pain of latency • We test end to end on almost every action WebDriver tests can catch things other levels of tests can’t • Multi-platform consistency • DOM consistency • Rendering issues • Catching inconsistency between app layers
  5. 5. UI Testing Portfolio • 50,000 unique Selenium tests • 7,500 unique WebDriver tests • full “browser compatibility”: IE6-IE11, Firefox, Chrome, Safari • some can run in headless browser like Phantom JS • 2,000 javascript unit tests • no DOM required • 10ms average execution time
  6. 6. 1,000,000 browser tests per day • Click here to add bullets
  7. 7. The Selenium Infrastructure ● Automatically creating, assigning, closing and reopening bug reports for test failures and flapping tests . ● Test jobs must meet failure thresholds (99%+). Code lines are locked for teams under thresholds ● Support for all combinations of OS & Browser using portable VM Templates, creating deterministic and identical test environments
  8. 8. Issues of scale • Assigning test failures to the responsible engineer • Non deterministic tests (flappers) and re-runs • Selenium jar upgrades are challenging • Browser upgrades are challenging • IT and automation ops provide different versions • Large suites: adjust run frequency, deprecate old tests
  9. 9. Salesforce UI Testing Pattern • Click here to add bullets
  10. 10. Proper Page Objects Encapsulate Selenium • Make an API of your page for your test • No imports for “openqa” or “selenium.org” in your test • All references to driver & selenium in the page object • Fail fast - call the page object factory in test setup • No assertions in page objects (or any test utility) • Share page objects across engineering teams • developer complete UI should have a page object • if you can’t make one it’s not a testable page • dependency challenges at scale
  11. 11. Challenges for Mobile Platforms • Adapting page objects to platform specific UI elements • Screen sizes and adaptive layouts • “stage left” - switching into contained apps • lightweight page objects = widgets • Avoid browser or platform specific code in actual tests • establish context elsewhere in utils or page objects • Automated tests in emulators (99%+) • manual tests on devices before release • exception: testing video
  12. 12. Sagar Wanaselja Lead Engineer swanaselja@salesforce.com
  13. 13. Problematic WebDriver Test
  14. 14. Better WebDriver Test
  15. 15. PageObject WebDriver Test
  16. 16. Diagnosing Test Failures • Screenshot and HTML capture at failure point • AssertionError or RuntimeException • Differences between localhost and automation VM • Optimistic timing assumptions • javascript, rendering, server response, etc. • Defensive “negative” testing • Getting to 100% tests passing
  17. 17. Pro Tips • Version tests and app code (and configuration) together • Use the Page Object model • Use the API to set up the test, not the UI • Anticipate Selenium and Browser releases • Allocate capacity to browsers most used by customers • log the user agent on every page view
  18. 18. More Pro Tips • Avoid using xpath selectors • Complex DOM makes xpath evaluation much slower • Some xpaths are not properly evaluated in WebDriver • When working with popups make sure you return focus to the main window • For static/simple HTML pages consider the HtmlUnitDriver
  19. 19. Mature Quality Strategies • Selenium for dynamic security vulnerability analysis • Pixel perfect image diffs • Accessibility testing (tab completion) • Testing javascript with unit tests • Selenium is for testing browser compatibility and DOM

×