1.
@wakaleo 1
A fresh look at test automation strategies
The Testing Pyramid Heresy
John Ferguson Smart http://johnfergusonsmart.com

2.
@wakaleo
John
Ferguson
Smart
John Ferguson Smart
“I help teams of smart people
learn to work together more efficiently,
to deliver better software sooner”
2
https://johnfergusonsmart.com

3.
https://johnfergusonsmart.com
“Test Automation is like any other tool
It’s either a benefit or a hazard”
3

4.
@wakaleo
https://johnfergusonsmart.com
How much are your tests worth?
4
Weekly cost
vs
weekly savings
$0
$20,000
$40,000
$60,000
$80,000
$100,000
$120,000
$140,000
$160,000
$180,000
1/09/2015 1/10/2015 1/11/2015 1/12/2015
Test0suite0cost0vs0savings0(per0week)
Cost0saved0par0week Test0suite0cost0per0week
Team cost is constant
Time saved increases with the
number of executed tests

5.
@wakaleo
https://johnfergusonsmart.com
How much are your tests worth?
5
$0
$200,000
$400,000
$600,000
$800,000
$1,000,000
$1,200,000
1/09/2015 1/10/2015 1/11/2015 1/12/2015
What0is0your0test0suite0worth?
Test0suite0ROI Test0suite0cost
Cumulated cost
vs
cumulated savings
Cumulated value should increase
faster than cumulated cost

6.
@wakaleo
https://johnfergusonsmart.com
Now factor in high maintenance
6
20-40%
Typical maintenance overhead
for poorly-written test suites

7.
@wakaleo
https://johnfergusonsmart.com
Now factor in high maintenance
7
Number of tests
Maintenanceoverhead

8.
@wakaleo
https://johnfergusonsmart.com
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
$400,000
$450,000
$500,000
1/09/2015 1/10/2015 1/11/2015 1/12/2015 1/01/2016
What0is0your0test0suite0worth?
Test0suite0cost Test0suite0ROI
Now factor in high maintenance
8
High-maintenance costs
reduce the number of
tests a team can write
High maintenance costs can
cause your test suite to lose value

9.
@wakaleo
https://johnfergusonsmart.com
Enter the Testing Pyramid
9

10.
@wakaleo
https://johnfergusonsmart.com
…or the Testing Ice-cream cone
10

11.
@wakaleo
https://johnfergusonsmart.com
In other words…
11
Don’t use a web test where a service or
unit test could do the same job.

12.
12

13.
13
“All models are wrong,
but some are useful.”
- George Box

14.
14

15.
15
The Testing Pyramid is about how a test is implemented.
But is this the best way to classify our tests?

16.
@wakaleo
Developer TestsQA TestsBusiness Tests
Tests come from different sources
16

17.
@wakaleo
Developer TestsQA TestsBusiness Tests
Tests come from different sources
17

18.
@wakaleo
Developer TestsQA TestsBusiness Tests
Tests come from different sources
18

19.
@wakaleo
Developer TestsQA TestsBusiness Tests
Tests come from different sources
19

20.
@wakaleo
DemonstrateDescribeDiscover
Tests have different roles
20

21.
@wakaleo
DemonstrateDescribeDiscover
Tests have different roles
21

22.
@wakaleo
https://johnfergusonsmart.com
Test (activities) that discover
22
Example Mapping
Impact Mapping
Feature Mapping

23.
@wakaleo
DemonstrateDescribeDiscover
Tests have different roles
23

24.
@wakaleo
https://johnfergusonsmart.com
Test (activities) that describe
24

25.
@wakaleo
https://johnfergusonsmart.com
Test (activities) that describe
25

26.
@wakaleo
DemonstrateDescribeDiscover
Tests have different roles
26

27.
@wakaleo
https://johnfergusonsmart.com
Example and
Feature Mapping
Tests have different roles
27
Discover
D
escribe
Dem
onstrate
Impact Mapping
Executable
specifications
Performance tests
Functional tests
Unit tests
Regression tests

28.
28
A new testing quadrant

29.
29
A new testing quadrant

30.
30
A new testing quadrant

31.
31
A new testing quadrant

32.
32
A new testing quadrant

33.
33
Goldilocks Test Suites

34.
@wakaleo 34
Questions?
John Ferguson Smart http://johnfergusonsmart.com