A presentation about tests antipatterns. What are typical problems connected with testing. Why and how we should avoid slow tests, obscure tests, overusing mocks, manual tests, fragile tests, erratic tests, eager tests, verbose tests, mystery guests in tests, test code duplication and high test maintenance cost. And how can we write tests clearly in Behavior-Driven-Development (BDD) style.
9. Slow Test - stabilizing a very slow
test
Don't steer a rover on Mars from Earth
Execute a test ...
8 minutes later: failure
Fix & rerun
10 minutes later: failure
Fix & rerun
10 minutes later: failure
10. Slow Test - stabilizing a very slow
test
What to do in the meantime?
GTD: don't multitask
CPU cache vs "brain cache"
Context switch cost
11. Slow Test - stabilizing a very slow
test
What to do in the meantime?
Solution: use the fastest machine
Have the machine in the company
Other tests on slower machines
12. Slow Test - stabilizing a very slow
test
What to do in the meantime?
For performance or stress tests with big
amount of data
Solution: use smaller data sets for
stabilization first
Then run with bigger datasets
13. Slow Test - stabilizing a very slow
test
Solution: use REPL environment to
stabilize it faster
Real-eval-print loop
A failure shouldn't invalidate previous results!
14. Slow Test - stabilizing a very slow
test
Solution: stabilize parts separately
Given a test:
Generate a big file.
Consume it.
Don't remove the file if failure occurs in (2).
(to be continued)
15. Slow Test
Stabilizing a very slow test
Individual slow tests
Executing all tests is too long
16. Slow Test - individual tests are
slow
What is a slow test?
a developer gets irritated or bored waiting for
finish
is tempted to get a break, surf the Internet,
chat, or walk around
17. Slow Test - individual tests are
slow
What is a slow test?
Sometimes more than a second, sometimes a
millisecond
Not long individually, but all tests execution is
too long
Or a group of tests is too long
18. Slow Test - individual tests are
slow
Impact:
Developers stop running them after every code
change
Developers wait for a coffee break, lunch or a
meeting to run them
Delayed feedback, loss of "flow"
Decreases productivity and job satisfaction
19. Slow Test - individual tests are
slow
Cause: Slow dependent component (1)
Solution: Fast Fake Objects
Real DB → Fake DB (e. g. In Memory DB)
Real Cloud → In memory cloud
Filesystems: HDD, SSD → RAM disk:
mount t tmpfs o size=200m tmpfs /mnt/tmp
20. Slow Test - individual tests are
slow
Cause: Slow dependent component (2)
Example: Backup applications
Solution: Recorded test
Record operations done by backup
applications
Warning: Recorded test may be a Fragile Test
21. Slow Test - individual tests are
slow
Cause: Slow environment or layer
Solution: Separate tested logic from
environment:
Hardware/GUI/HTTP layer
Threads, processes
Clouds, etc.
Kernel code
22. Slow Test - individual tests are
slow
Cause: Over-engineered fixture
Long setup of many complicated objects
Even if most of them are not needed
Or using full product in tests
Solution: General Fixture → Minimal
Fixture
23. Slow Test - individual tests are
slow
Cause: Manual tests
Slow by definition
Even slower if repeated
"Please check that the system just works"
before a release
Automated tests may get written anyway later
24. Slow Test - individual tests are
slow
Cause: Manual tests
Often under pressure or under "give-it-to-me-
now driven-development"
No future returns
May be worthless after just a single commit
Lack of tests automation will slow test
development later
25. Slow Test - individual tests are
slow
"Just Say No to More End-to-End Tests"
Testing pyramid
Manual tests
End-to-end tests
Integration tests
Component tests
Fast Unit Tests
26. Slow Test
Stabilizing a very slow test
Individual slow tests
Executing all tests is too long
27. Slow Tests - executing all tests is
too long
In some big IT companies:
build time: one minute
release to production: 8 minutes
not just a release – a release to production!
What problems may it address?
28. Slow Tests - executing all tests is
too long
Impact: Developers context-switching,
worse productivity
tests start ...
15 min: a test fails
fix & rerun
20 min: a test fails
29. Slow Tests - executing all tests is
too long
Impact: Integrating code slower
Bigger release queue
Slower features delivery, missing deadlines
30. Slow Tests - executing all tests is
too long
Impact: Production big fixes are
delayed
A customer requires a fix, we need to test it
Release a quick fix or a well tested fix?
31. Slow Tests - executing all tests is
too long
Cause: Too much overlap between
tests
Test1: [ DATA LOADING ][doSth][check]
Test2: [ DATA LOADING ][doSth][check]
Test3: [ DATA LOADING ][doSth][check]
32. Slow Tests - executing all tests is
too long
Cause: Too much overlap between
tests
Use Shared Fixture?
Shared setup for many tests
Warning: may get closer to Erratic Tests,
High Test Maintenance Cost
33. Slow Tests - executing all tests is
too long
Cause: Too Many Tests
Too many tests run too frequently
Solution: subsets of tests can be run at
the time
All tests within larger period of time
Prerelease tests: all with fast fakes
Postrelease tests: with real objects
34. Slow Tests - executing all tests is
too long
Cause: Too Many Tests
The system is too large
Solution: break it into independent
subsystems
35. Slow Tests - executing all tests is
too long
Cause: Slow Machines
Only one integration server
Too slow integration servers
Insufficient parallelization: look at "top",
"vmstat"
38. Overuse of mocks
Impact: Tests are harder to maintain
Focus: implementation vs public interface
Focus: implementation vs behavior
Extra code for mock behavior needed
Over-specified software
Fragile
39. Overuse of mocks
Impact: Less assurance that the code
is working properly
It's hard to guarantee that the mocks behave as
real implementations
Especially over time
40. Overuse of mocks
When the real object cannot be used:
They are too slow
Their setup is too complex
They charge money, etc.
45. Overuse of mocks
We actually don't care for interaction with cash
service and tax service here
46. Overuse of mocks
Alternative: use a fake
def should_compute_doubled_price_for_two_cups():
t = TeaHouse(
FakeInMemoryCashService(), FakeTaxService()
)
c = CreditCard(money=usd(30))
t.set_tea_price(usd(10))
t.order_cup(c)
t.order_cup(c)
assert t.money_total == usd(20)
47. Overuse of mocks
Alternative: use a stub or a non-strict
mock:
def should_compute_doubled_price_for_two_cups():
t = TeaHouse(
StubCashService(always_ok=True), StubTaxService()
)
c = CreditCard(money=usd(30))
t.set_tea_price(usd(10))
t.order_cup(c)
t.order_cup(c)
assert t.money_total == usd(20)
48. But in this particular
situation, even better
is ...
54. Obscure Test
Impact:
Harder to understand
Harder to maintain
Developers will not read them as
documentation
May lead to Can result in a Buggy Test, and
then Production Bugs
56. Obscure Test
Is it a good test?
def should_have_the_same_permissions_after_copy():
fs = FileSystem("rw")
f = File("a.txt", "rwxrwxrwx")
fso = FileSystemOps(fs)
fso.copy(f, "b.txt")
assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))
57. Obscure Test
Is it a good test?
def should_have_the_same_permissions_after_copy():
fs = FileSystem("rw", "noauto", 620, false,
true, true, 8096, 10*1024*1024*1024, 1000000)
f = File("a.txt", "rwxrwxrwx", 64, "xyzxyz",
"20161212", "20161212", "20161212")
fso = FileSystemOps(fs, true, false, 1024, 11)
fso.copy(f, "b.txt")
assert(fso.fileperm("a.txt") == fso.fileperm("b.txt"))
58. Obscure Test
Cause: Irrelevant information
which values affect the outcome?
There can be a hundred constants in a test
code
59. Obscure Test
Cause: Too much information
Solution: introduce higher-level test
language
def file_copy_permissions_should_be_as_original():
given_a_filesystem()
given_a_file_with_permissions("rwxrwxrwx")
when_a_file_is_copied()
then_the_new_file_has_permissions("rwxrwxrwx")
60. Obscure Test
Cause: Too much information, Eager
Test
The test verifies too much functionality in a
single
Test Method and therefore is hard to
understand
62. Obscure Test
Cause: Too much information, Eager
Test
Often a Fragile Test
And a High-Maintenance Test
63. Obscure Test
Cause: Too much information
Not caring about clean code
No refactoring
"These are only tests"
"just do it in-line" mentality
64. Obscure Test
Cause: Too little information – Mystery
Guest
Cause and effect not clear
Something is hidden from the Test Method
65. Obscure Test
Cause: Too little information – Mystery
Guest
Doesn't exist because is copied to root?
def test123():
fs = createFileSystem()
f = new File("a.txt", "rwxrwxrwx", 64*MB, "xyzxyz",
"20161212", "20161212", "20161212"
)
fso = new FileSystemOps(fs, true, false, 1024, 11)
fso.copy(f, "/b.txt")
assert(fso.exists("b.txt") == false)
66. Obscure Test
Cause: Too little information – Mystery
Guest
No, because createFileSystem creates read-only
filesystem:
def createFileSystem():
return new FileSystem(
"ro", "noauto", 620, false, true, true,
8096, 10*1024*1024*1024, 1000000
)
)
67. Obscure Test
Cause: Setup-action-verify phases not
clear
def test123():
fs = createFilesystem()
f = createFile("a.txt")
fso = createFso()
f2 = fso.copy(f, "b.txt")
s1 = fso.size("a.txt")
s2 = fso.size("b.txt")
assert(s1 == s2)
Do we test "size" here or "copy"?
Solution: Given-when-then
68. Obscure Test
Cause: Setup-action-verify phases not
clear
Solution: Given-when-then:
given_filesystem()
given_file()
given_file_system_operations()
when_the_file_is_copied()
then_the_new_file_size_is_as_original()
70. Obscure Test
Cause: Overcomplicated Fixture
The fixture is too big and too complex
To understand the test, reading the fixture may
be necessary
Solutions:
Divide a General Fixture into smaller Fixtures
Use Minimal Fixture
Use Facades
71. Obscure Test
Cause: Indirect Testing
Testing one component through other
components
Example: Execute a product in debug mode to
see if log rotation works
Root Cause: software not designed for
testability
72. Sources
Gerard Meszaros - xUnit Test Patterns
testing.googleblog.com
Paul Chiusano and Rúnar Bjarnason - Functional
Programming in Scala