This document discusses how eBay tests its products and services. It notes that testing is done globally by over 300 test engineers across multiple sites. While manual testing remains important, especially for new features, eBay has invested heavily in automation to help test over 3500 test cases across its European sites in under an hour. This large-scale automation was made possible through the use of a testing grid with over 150 virtual machines and browsers. The document shows that increased automation efforts have correlated with reduced bug rates and higher quality bugs found by the EU testing team.

1. How we test at eBay
Michael Palotas
Head of Quality Engineering Europe
Zürich, 17 August 2011

2. The Company
2
Produced by: Michael Palotas

3. eBay Inc.
3
Produced by: Michael Palotas

4. Setup
Mobile
Global Portland Global
Product USA Product
San Jose Chennai
USA India
EU
Quality
Engineering
Global Local
Product
(Zürich, London,
Berlin, Paris)
Product
Shanghai Zürich,
China London
Adjacencies
(i.e. Brands
for Friends)
4
Produced by: Michael Palotas

5. Not only the Web
Windows
Desktop
Web
Mobile
5
Produced by: Michael Palotas

6. Landscape
Teams all around the globe in all possible time zones
90.000 development days / year
ina
India + Ch
our ced to
ece s outs
M ajor pi
300 test engineers around the globe
nt
t environme
tes
Unstable
Continuou 1000+ new bugs found every week
s Integratio
n not cons
istent
eBay Inc Produced by: Michael Palotas
6

7. Waterfall...SCRUM…and everything in between
7
Produced by: Michael Palotas

8. Testing at eBay in the past
Very manual
Manual regression testing for 22 sites, every 2 weeks
Teams did not rely on automation
Skill set of QA Engineers mainly non-technical
Sources: www.linesofflight.net
www.generallyAwesome.com
8
Produced by: Michael Palotas

9. Testing Today – Ownership of Quality
Manual Test
Consulting
+
Coaching
Automation
9
Produced by: Michael Palotas

10. Manual Test
Still very important, mainly exploratory type
All new features are tested manually first
Edge cases
Not automatable tests
Manual testing still yields the highest number of bugs
10
Produced by: Michael Palotas

11. Automation
Automation is not about eliminating jobs !!!
Save time and reinvest into manual testing
Regression Testing (fully) automated
Programmatic approach (JAVA)
Open source only (WebDriver/Selenium 2 GRID)
3500 end to end testcases running in less than 1 hour
11
Produced by: Michael Palotas

12. GRID
eBay Europe:
- 3500 e2e regression testcases
- 2-3 minutes execution time (each)
executions
Execution time 7000 minutes (~116 hours = ~5 days)
Sequential
Parallel
Execution Test
1
Test
2
Test
3
Test
…
Test
…
Test
3500
Execution Time
Parallel Test Test Test Test Test
Parallel executions
150 Virtual Machines / Browsers in
execution in Test Test
the WebDriver GRID
Test Test Test
Execution time: 60 minutes
Grid
Test Test Test Test Test
Execution Time
12
Produced by: Michael Palotas

13. Coaching / Consulting
EU team conducts project audits
Improvement initiatives
Consulting for dev and test teams
13
Produced by: Michael Palotas

14. Software Measurement - SONAR
14
Produced by: Michael Palotas

15. Project Audits
15
Produced by: Michael Palotas

16. EU Quality Engineering Team – Automation Impact
Team % of global Bugs 2009
" Team
makes up for 20.00%
18.00% Start of
only 2% of global test 16.00% automation
efforts in
engineer headcount 14.00%
early 2009 Contribution
12.00% Trendline
10.00%
8.00%
6.00%
" Clear correlation 4.00%
between global bug 2.00% headcount share
contribution and .00% 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13 1 2 3 4 5 6 7 8 9 10 11 12 13
Week
automation
investment / coverage 18.0% Team % of global Bugs 2010
16.0%
14.0%
12.0%
10.0%
8.0%
6.0%
4.0%
headcount share
2.0% headcount share
0.0% 16
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38Palotas 43 44 45 46 47 48 49 50 51
Produced by: Michael 39 40 41 42
Week

17. Bug Quality
Bug Quality = real bugs / submitted bugs
100
Bug Quality in % (2008-2010)
95
90
Start of
automation
efforts
85 Bug Quality 2008-2010
80
75
Q1 08 Q2 08 Q3 08 Q4 08 Q1 09 Q2 09 Q3 09 Q4 09 Q1 10 Q2 10 Q3 10 Q4 10
17
Produced by: Michael Palotas

18. eBay checkbox – vs. HTML checkbox
<div align="center”>
<input type="checkbox" name=”myName" value=”eBay"> eBay<br>
</div>
18
Produced by: Michael Palotas

19. Summary
Testing is changing
Great career opportunities in testing
More technical skill sets required
Testautomation can have a big impact
….
or make your life hell
Essential to work with developers
Measure and improve
19
Produced by: Michael Palotas

20. Contact: mpalotas@ebay.com
THANK YOU!
QUESTIONS?
20
Produced by: Michael Palotas