Mutation-Testing mit PIT - wie gehts, was kanns, was bringts, was kostets

1. e-movimento Software Design & Beratung GmbH
1030 Wien ● Marxergasse 7/26 ► www.e-movimento.com
Mutation-Testing mit PIT
www.pitest.org

2. Mutation Testing – Was?
 Definition (https://en.wikipedia.org/wiki/Mutation_testing):
Mutation testing (or Mutation analysis or Program
mutation) is used to design new software tests and evaluate
the quality of existing software tests. Mutation testing
involves modifying a program in small ways. Each mutated
version is called a mutant and tests detect and reject
mutants by causing the behavior of the original version to
differ from the mutant. This is called killing the mutant. Test
suites are measured by the percentage of mutants that they
kill.
► www.e-movimento.com, Sebastian Dietrich2

3. Mutation Testing – Wozu?
 Ziel:
The purpose is to help the tester develop effective tests or
locate weaknesses in the test data used for the program or
in sections of the code that are seldom or never accessed
during execution. Mutation testing is a form of white-box
testing.
 Reicht das?
"Quis custodiet ipsos custodes?“
“Wer wird die Wächter selbst bewachen?“
[Juvenal, Satire 6, 347f.]
► www.e-movimento.com, Sebastian Dietrich3

4.  muJava A mutation tool for Java that
includes class-level operators
 mutate.py A Python script to mutate
C-programs
 Mutator A source-based multi-
language commercial mutation
analyzer for concurrent Java, Ruby,
JavaScript and PHP
 Bacterio Mutation testing tool for
multi-class Java systems
 Javalanche Bytecode-based mutation
testing tool for Java
 Major Compiler-integrated mutation
testing framework for Java
 Jumble Bytecode-based mutation
testing tool for Java
 PIT Bytecode-based mutation
testing tool for Java
 Stryker Mutation testing tool for
JavaScript
 Mutant AST based mutation testing
tool for Ruby
 Jester Source-based mutation testing
tool for Java
 Judy Mutation testing tool for Java
 Heckle Mutation testing tool for Ruby
 NinjaTurtles IL-based mutation testing
tool for .NET and Mono
 Nester Mutation testing tool for C#
 Humbug Mutation testing tool for PHP
 MuCheck Mutation analysis library for
Haskell
Mutation Testing - PIT
► www.e-movimento.com, Sebastian Dietrich4

5. PIT
 Ändert Bytecode und lässt
Tests laufen
 Arbeitet mit ant, maven,
gradle, ...
 Plugins für Eclipse, IDEA, ...
 Dunkelgrün = Mutation Coverage
 Hellgrün = Line Coverage
 Hellrot = keine Line Coverage
 Dunkelrot = keine Mutation Coverage
► www.e-movimento.com, Sebastian Dietrich5

6.  Conditionals Boundary:
<  <=, >=  >, ...
 Negate Conditionals:
==  !=, >=  <
 Remove Conditionals:
a == b  true
 Math Mutator:
+  -, &  |, <<  >>, ...
 Increments Mutator:
i++  i—, i--  i++
 Invert Negatives:
-i  i
 Inline Constant:
int i = 5;  int i =-1;
 Return Values:
return true;  return false;
return x;  return x+1;
return obj;  return null;
return null;  RuntimeExc
 Void Method Call:
doSomething();  ;
 Non Void Method Call:
return5();  0;
 Constructor Call:
new Object();  null;
PIT – Mutators (Auszug – in Summe 7/13)
► www.e-movimento.com, Sebastian Dietrich6

7. Praxis – UnitTests eines Moduls
 1.735 Tests  69,5% Coverage (lt. Sonar)
 Min. 5GB RAM für Maven-Build
  12857 Mutationen, davon 5312 gekillt (weil doppelt)
  19.117 Tests
  41% Mutation Coverage (63% Line Coverage)

8. Laufzeit
 mvn test  3:18 min.
vs.
 mvn pitest-maven:mutationCoverage 
► www.e-movimento.com, Sebastian Dietrich8

9. Laufzeit – Aber...
 Incremental analysis möglich (nicht getestet):
 Mutation Killed, weder Test noch Klasse geändert  Mutation gilt
weiterhin als gekilled
 Mutation Überlebt, weder Test noch Klasse geändert, kein neuer
Test covered diese Klasse  Mutation gilt weiterhin als überlebt
 Mutation Killed, Klasse oder Test geändert  Mutation wird
vermutlich weiterhin gekilled werden  höhere Prio
 ...
► www.e-movimento.com, Sebastian Dietrich9

10. Konklusio
 Für Coverage-Kennzahlen größerer Module ungeeignet
(da zu viel RAM nötig)
 Für Build vermutlich ungeeignet (da zu langsam –
incremental analysis check noch offen)
 Zum Schreiben/Perfektionieren/Verbessern einzelner
Unit-Tests gut geeignet
 Am besten mit Plugins arbeiten
► www.e-movimento.com, Sebastian Dietrich10