7. Why doesn’t unit
testing work?
We can’t test things of business
value
Not knowing what to test, we test
many things that don’t need testing
Automated crap is still crap
8. Unit tests can’t test
things of business value
Can you demonstrate that Method X of
Object Y will ever be invoked in practice?
If a unit test fails, how does it affect ROI?
What’s amazing is that we continue to test…
Paths that will never be exercised
Methods that are never used
Naive method invocation sequences
9. Most unit tests are
unnecessary
We test scenarios that will never be
exercised in the real world
We test methods that will never be
invoked
We supply data that a real application
would never have to process
10. Automated crap is
still crap
Design requires intelligence:
intelligence cannot be automated
Autonomation instead of automation
Testing is about exploratory inference
Get manual procedures working first
and then automate the mundane parts
21. Weinberg’s Law of
Decomposition
Wasted testing effort
Use
Size
Test
Size
22. Part III: You Will Have Bugs!
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
23. Testing waste
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
24. Testing waste
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
Type 1
Muda
Type 1: Defects
25. Testing waste
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
Type 2
Muda
Type 1
Muda
Type 1: Defects
Type 2: Stuff
customers
don’t ask for
26. Testing waste
Scenarios we
test in the lab
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
Type 2
Muda
Type 1
Muda
Type 4
Muda
Type 1: Defects
Type 2: Stuff
customers
don’t ask for
Type 4:
Overprocessing
27. Testing waste
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
Type 1
Muda
Type 4
Muda
Type 1: Defects
Type 2: Stuff
customers
don’t ask for
Type 4:
Type Overprocessing
1 Muda
(field)
28. Testing waste
Removing tests
leaves field faults
Scenarios
exhibiting
faults
Scenarios
invoked in
the field
undetected!
Type 1
Muda
Type 4
Muda
Type 1: Defects
Type 2: Stuff
customers
don’t ask for
Type 4:
Type Overprocessing
1 Muda
(field)
29. Why do we unit
test?
Because we can
It doesn’t require teamwork
We don’t have to talk with the business
We get to interact with tools instead of
people
Our decisions are emotional, not rational
30. Reduction of Waste
Type 1: Better testing (e.g., going beyond
contextual testing to exploratory & experience-based
testing) (and better yet, better analysis
and design, where the real payoffs are)
Type 2: Don’t test dead code
Type 4: Remove redundant and tautalogical
tests, and tests that give no information
Type 1 in the field: Ship your tests!
31. Don’t take the tests
out when you ship!
Remember, you will have bugs in the
field!
Deployment is your largest test driver
Ship your tests as part of the code
IN case of a failure, it saves you
from Type 9 muda (unsafe execution)
33. How Adding ASSERTS reduces work
2500000
2000000
1500000
1000000
500000
0
15 bugs p=0.01 15 bugs p=0.02
10 bugs p=0.01 10 bugs p=0.02
5 bugs p=0.01 5 bugs p=0.02
0 1 2 5 10 20 50 100
number of ASSERTS
Work, Tsinglechecks
Overload
Magazine,
October 2014
34. Defensive
Programming
… is largely the incorporation of
tests in the code
Long known for high-reliability
systems
The value of asserting the obvious
Design by Contract
35. Assertions make code
readable!
- (void) prepareForSegue: (UIStoryboardSegue*) segue
sender: (id) sender
{
assert(segue != nil);
if ([[segue identifier] isEqualToString:
@"SegueAvatarPickerToPickOneLoginPlayer"]) {
assert((UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPhone));
// The field parentViewController_ (instance variable) is
// maybe null if we are in iPhone mode and doing a real
// segue, instead of a popover as we do on the iPad. So
// get the right object from the segue record.
KnowsyPlayerPickerCommonViewController *parentViewController =
[segue destinationViewController];
[parentViewController setKnowsyData: knowsyData_];
} else {
// iPad - no special processing - should be no popover
assert(parentViewController_ == nil);
}
}
36. Part IV: Beyond agile
testing, Lean testing
When a test finds a bug, it’s about
fixing the process — not the bug
The long-term payoff is higher!
No bug-tracking systems: we fix
problems NOW.
The tester should be the team’s
conscience
37. Other Lean-isms
Deming: Quality is inversely
proportional to the number of
testers
GM kept around its cars for 1 - 6
weeks for test and repair: Toyota
drove them off the end of the
assembly line onto the car carriers.
38. Conclusion
Be humble about what tests can accomplish
It’s not about testing, but value
Design tests to assess properties of business value
Go beyond unit testing to defensive
programming and design-by-contract
As agile testers, focus on process first and
product second