Devoxx Ignite 2014 session

1. KILLTHE MUTANTS
a better way to test your tests

2. SHOW OF HANDS
let's do a
• Unit testing
• Continuous integration
• Code coverage
• Mutation testing

3. CODE COVERAGE
• Measure the lines (or branches) that are executed during testing

4. CODE COVERAGE

5. QUIS CUSTODIET IPSOS CUSTODES?
Who watches the watchmen?

6. MUTATIONTESTING
• Proposed by Richard J. Lipton in 1971
• Surge of interest in the 1980s
• Time to revive this interest!

7. TERMINOLOGY: MUTATION

8. TERMINOLOGY: MUTANT
• A mutant is a mutated version of your class

9. MUTATIONTESTING
• Generate (a lot of) mutants of your codebase
• Run (some of) your unit tests
• Check the outcome!

10. OUTCOME #1: KILLED
• A mutant is killed if a test fails (detecting the mutated code)
• This proves the mutated code is properly tested

11. OUTCOME #2: LIVED
• A mutant didn’t trigger a failing test…

12. OTHER OUTCOMES
• TIMED OUT: Infinite loop
• NON-VIABLE: Illegal bytecode
• MEMORY ERROR: Out of memory
• RUN ERROR: Other…

13. TOOLING
• µJava: http://cs.gmu.edu/~offutt/mujava/ (inactive)
• Jester: http://jester.sourceforge.net/ (inactive)
• Jumble: http://jumble.sourceforge.net/ (inactive)
• javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive)
• PIT: http://pitest.org/

14. USING PIT
• Manipulates bytecode with ASM (at runtime)
• Mutants are never stored

15. MUTATORS
• Example of mutators:
> into >=
< into <=
>= into >
<= into <

16. MUTATIONTESTING IS SLOW?

17. WHICHTESTSTO RUN?
• PIT uses code coverage!

18. WHICHTESTSTO RUN?
• A mutation is on a line covered by 3 tests? Only run those.

19. EASYTO USE:
• Standalone Java process
• Build:Ant and Maven plugin
• CI: Sonarqube and Gradle plugin
• IDE: Eclipse and IntelliJ plugin

20. SUMMARY
• Code coverage: BAD
• Mutation testing: GOOD!
• Automatic, easy to add, and meaningful!