The document discusses ways to improve automated testing practices, such as by making tests easier to write, more readable, and more meaningful. It suggests using mocking frameworks to reduce test complexity, following standards and examples, and adopting a testing framework like Spock that addresses common issues. While progress has been made, there is still work to be done to encourage writing tests, make tests easy to write and understand, and ensure tests are meaningful and focused on the purposes of testing.
This is a story of how to get more out of your automated tests
This is my experiences at MongoDB, not JetBrains
We had tests, we had coverage, but it felt more like a box-ticking exercise than a commitment to the value of tests or testing.
And then we started using Spock, and magically everything got better
This is NOT one of those talks
Instead, I’m going to talk about the problems we were facing
And some of the solutions that helped us, yes including some technology choices
But we’ll see WHY they helped us
We know testing is important, yet we still don’t do it when we should
Common reasons why not
1 Too hard to setup
2 Single use code
3 The code is simple
4 Too hard to mock
5 UI code
6 Not enough time
7 Effort vs Value
8 I'm the only maintainer
Let’s take a step back and look at a different question
Me
Asking why Jenkins was failing
Fixing tests
Adding checkstyle
Fixing up checkstyle errors
Dishing out remaining errors
Pushing for tests in code reviews
Get things green
Require tests for code review
Got use to making sure Jenkins and travis were green
Got used to writing tests
Reports are useful too
There comes a point where it’s too hard to get the team to do everything you want
You want it to be easy and logical for others to know what to do and how to do it
The champion’s job is to put into place processes, practices, examples that become second nature to the team
- checkstyle a success
- unit testing not so much
I kept wanting to re-write other’s tests, to add more tests.
Good that our code review process didn’t allow that, or I would have become a tester
Champion and good habits are a start
Helped us address writing tests and writing readable tests to some extent
We still had problems
Some technical, some around improving quality (meaningfulness) of tests)
Insert Jenkins matrix?
MongoClientURITest
Most of all
It needs to be automatic for all the developers
Found out about it at a conference like this one
Was sceptical a library could solve what I thought were behavioural problems
Addresses quite a lot of the issues we had
Week of Spike
// String method names// given/when/then// Map syntax// Type coercing// new BasicDBObject("$set", new BasicDBObject("x", 2)// equals
Halfway
Two things
- @Subject
- “when”
Setup is explicit
When the setup section gets big, sign that we should refactor (not just a Spock solution)
Use “Fixture” for example – testDollarOutOnSecondary
Same problem, different side of the coin
- Catching server bugs
- Catching regressions
- Checking combinations of servers & drivers & java
- How the team embraced it – using Spock for integration and API-level tests
- Jenkins and Travis are now generally green, test coverage is up, tests written for new sections (async)
Not driven by me:
- Introduction of server specs
- Then test specs to prove a driver meets the criteria
- In future, would be great to have a TCK-like set of tests that can prove a driver meets MongoDB requirements (or some set of them)
- Catching server bugs
- Catching regressions
- Checking combinations of servers & drivers & java
- How the team embraced it – using Spock for integration and API-level tests
- Jenkins and Travis are now generally green, test coverage is up, tests written for new sections (async)
Not driven by me:
- Introduction of server specs
- Then test specs to prove a driver meets the criteria
- In future, would be great to have a TCK-like set of tests that can prove a driver meets MongoDB requirements (or some set of them)
Other problems
- Performance
- Dynamic lang has downsides
- Now have even more inconsistency in our tests (but at least we know which are the new ones)
- Refactoring – not so awesome with Spock
- Still not clear to people what the different tests are for – unit, functional, integration, acceptance
Perf is worse for running individual tests (Groovy has a startup cost)
Perf is slower to run the overall build (Groovy, plus adding an extra gradle step)
Perf is not RELIABLE – performance critical tests (i.e. with timeouts) needed to be Java
- Masks errors
- Finding problems at runtime instead of compile-time
- (On the other hand, this is really useful when you don’t care about types or nulls or whatever)
- Refactoring – not so awesome with Spock
Now we have old tests, good junit, spock (unit), spock (functional)
Expected spock to be used for unit only
Then we could be clear about spock – single class or a couple of classes
Now we’re not really sure about the difference between unit, functional, acceptance.
What I’d like it to do
- I still want tests that document the system
- I’d like automated perf tests, but production-like data is too hard
- I’d like soak tests
- I’d like much more complicated tests, which do more than one thing (is it not enough to test each piece individually?)
- I’d like more unit tests
- Basically want more types of tests, with better clarity on the purpose of each test type. Originally hoped Spock would be for unit, and all integration tests would be truly end-to-end and Java
- Pairing
- Code Reviews
- Make it simple
- Make it fun
- Understand WHY we test
- *Could* use coverage
Groovy helped us
- DSLs
- Gwen
- even (shock) comments
- good test names
- test one thing (have single reason for failure?)
- make it clear what’s under test, what’s expectations
- extract common functions (connect to DB, get server, etc)
- Have good example tests for people to follow (guidelines)
- Make tests a first class citizen (code review for example)
- This is all just common code practices, it’s the same for tests as it is for production code. Somehow, we fail to do this in tests
- This might actually be the hardest thing to do. Define meaning
- Code review
- Understand the purpose of the test – end-to-end? Unit? Checking the server? Checking your assumptions? Documentation? Requirements?
- Tester mindset – What if? What does this mean? What are the implications?
- |Good idea to get someone who isn’t close to the code or the system to check out the tes
Suspect coverage is the wrong metric to drive things
A new language might be a fun way to get developers to pay attention to tests (or it might be a hindrance)
Jenkins
Static analysis
Coverage
Either fix it
Or delete it
Have a champion – someone needs to give a shit
And they need to not get annoyed at pushing this all the time
Don’t be religious
Don’t hold tight to ideals, go with what works (Spock not just for unit; having to let go of acceptance tests)