Overview of Selenium, WebDriver, Watir and related open source cross-browser testing technologies. Presentation given by Martin Kleppmann, founder of browser testing service Go Test It, at Ruby Manor 2009.

1. Browser-level testing
Martin Kleppmann
Go Test It
Email: martin@go-test.it
Blog: yes-no-cancel.co.uk
Twitter: @martinkl

2. “But I have unit
tests!”

4. ?

5. Array literals?
var x = [1,];
x.length == 1 // Firefox 3.5
x.length == 2 // IE 7

6. Uninitialised vars?
<div id="hello"></div>
typeof(hello) == 'undefined' // Firefox 3.5
typeof(hello) == 'object' // IE 7

8. Attempt #1:
Simulate events in JavaScript.
(Selenium, Windmill, Sahi)

9. Attempt #1: Simulate events in JavaScript.
Selenium example.com

10. Attempt #1: Simulate events in JavaScript.
click
Selenium example.com

11. Attempt #1: Simulate events in JavaScript.
click
Selenium example.com
http://localhost:8080 http://test.example.com

12. Attempt #1: Simulate events in JavaScript.
Cross-domain
security policy
click
Selenium example.com
http://localhost:8080 http://test.example.com

13. Attempt #1: Simulate events in JavaScript.
Selenium example.com

14. Attempt #1: Simulate events in JavaScript.
Selenium example.com
test.example.com proxy

15. Attempt #1: Simulate events in JavaScript.
Selenium example.com
test.example.com proxy
/selenium-server/ /*

16. Attempt #1: Simulate events in JavaScript.
click
Selenium example.com
test.example.com proxy
/selenium-server/ /*

17. Limitations
• Redirect to other domain or https
• Outside of DOM: pop-ups, Flash, ...
• Slow

18. Attempt #2:
Browser-specific automation
APIs.
(WebDriver, Watir)

20. IE
Firefox
Safari
Chrome
Opera
HTMLUnit

21. IE
Firefox
Safari
WebDriver
Chrome
Opera
HTMLUnit

22. IE
Firefox
Safari
WebDriver
Chrome
Opera
HTMLUnit

23. IE
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

24. IE
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

25. IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

26. IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

27. IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

28. Selenium WebDriver Watir
IE
Firefox
Safari
Chrome
Opera
HTMLUnit

29. Selenium WebDriver Watir
IE HTA/Proxy ✔ COM ✔ COM
Firefox Ext/Proxy ✔ Extension ✔ Extension
Safari Proxy ✘ ✔ AppleScript
Chrome Proxy ✔ Extension ✔ V8 Debug
Opera Proxy Dragonfly? ✘
HTMLUnit ✘ ✔ ✔ Celerity

30. Watir
require 'watir'
Watir::Browser.default = 'firefox'
describe 'Google' do
before(:each) { @browser = Watir::Browser.new }
after(:each) { @browser.close }
it 'should return search results for "hello world"' do
@browser.goto "http://www.google.co.uk"
@browser.text_field(:name, "q").set("hello world")
@browser.button(:name, "btnG").click
@browser.contains_text(
"Hello world program - Wikipedia").should be_true
end
end

31. WebDriver
require 'selenium-webdriver'
describe 'Google' do
before(:each) { @browser = Selenium::WebDriver.for :firefox }
after(:each) { @browser.quit }
it 'should return search results for "hello world"' do
@browser.navigate.to "http://www.google.co.uk"
@browser.find_element(:name, "q").send_keys("hello world")
@browser.find_element(:name, "btnG").submit
@browser.find_element(:partial_link_text,
"Hello world program - Wikipedia")
end
end

32. Selenium
gem 'selenium-client'; require 'selenium/client'
describe 'Google' do
before(:each) {
@browser = Selenium::Client::Driver.new :browser => "*firefox",
:url => "http://www.google.co.uk"
@browser.start
}
after(:each) { @browser.stop }
it 'should return search results for "hello world"' do
@browser.open "/"
@browser.type "q", "hello world"
@browser.click "btnG", :wait_for => :text, :text => 'Results'
@browser.is_text_present("Hello world program - Wikipedia").
should be_true
end
end

33. Go Test It (≈Selenium)
require 'gotest'
gotest "http://www.google.co.uk" do
open "/"
type "q", "hello world"
click "btnG"
verify_text_present "Hello world program - Wikipedia"
end

34. Cucumber
Feature: Google search
In order to learn something new
As an enthusiast for obscure programming languages
I want to find "Hello world" programs on Google
Scenario: Search for "hello world"
Given that I am on google.co.uk
When I enter "hello world" into the search box
And I submit the form
Then I should be shown a list of results
And the page should contain "Hello world program -
Wikipedia"

35. IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

36. IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

37. IE
Selenium
Firefox
Safari
WebDriver
Chrome
Opera
Watir
HTMLUnit

38. The future? IE
Selenium
Selenium API
Firefox
Safari
WebDriver
Chrome
Opera
Watir
Watir API
HTMLUnit

39. Ok.

40. Fundamental goal:
Robust tests

41. Choose your locators
carefully

42. Choose your locators
carefully
id = searchInput
name = search
xpath = //form[@id='searchform']/input[2]

45. link = EU makes 7bn euro climate pledge

46. link = EU makes 7bn euro climate pledge
xpath = //td[@class='text']/div[1]/a

47. link = EU makes 7bn euro climate pledge
xpath = //td[@class='text']/div[1]/a
css = a.tsh

48. People are complicated
Frontend tests are
messy

49. 2 approaches:
Disposable tests
(record/playback, quick&dirty)
Domain-specific abstractions
(carefully crafted)

50. Which
approach will
you take?

51. Thank you!
Martin Kleppmann
Go Test It
Email: martin@go-test.it
Blog: yes-no-cancel.co.uk
Twitter: @martinkl

52. Fundamental goal:
Parallel tests

53. Don’t assume a clean database

54. Don’t assume a clean database
Unit tests:
Load fixtures
Given...
When...
Then...
Reset database

55. Don’t assume a clean database
Unit tests: Functional tests:
Load fixtures Set up example data
Given... Do stuff Do stuff
When...
Check Check
Then... outcome outcome
Reset database

56. Tolerate partially run tests

57. Tolerate partially run tests
(Assume there is no frob)
Create frob
Check frob was created
Delete frob

58. Tolerate partially run tests
(Assume there is no frob)
Create frob
Check frob was created
Delete frob

59. Tolerate partially run tests
Bad:
Boom!
(Assume there is no frob)
Create frob
Check frob was created
Delete frob

60. Tolerate partially run tests
Bad: Better:
Boom!
(Assume there is no frob) Delete frob if it exists
Create frob Create frob
Check frob was created Check frob was created
Delete frob Delete frob (optional)

61. Avoid race conditions: isolation

62. Avoid race conditions: isolation
Delete frob if it exists Delete frob if it exists
Create frob Create frob
Check frob was created Check frob was created

63. Avoid race conditions: isolation
Log in as User A Log in as User B
Delete frob if it exists Delete frob if it exists
Create frob Create frob
Check frob was created Check frob was created

64. Avoid race conditions: isolation
Log in as User A Log in as User B
Delete frob if it exists Delete frob if it exists
Create frob Create frob
Check frob was created Check frob was created
User A and User B: independent & not used concurrently

65. Make test runs unique

66. Make test runs unique

67. Make test runs unique

68. Take care with external services

69. Thank you!
Martin Kleppmann
Go Test It
Email: martin@go-test.it
Blog: yes-no-cancel.co.uk
Twitter: @martinkl