9. @DevPaco
Unit test example
public class Calculator {
public int multiply(int a, int b) {
return a * b;
}
}
class CalculatorTest {
@Test
void testMultiply() {
assertEquals(20, new Calculator().multiply(4, 5));
}
@Test
void testMultiplyWithZero() {
assertEquals(0, new Calculator().multiply(0, 5));
}
}
16. @DevPaco
• Projects evolve, grow
• Tests evolve, sometimes missed when refactoring
• Test code often not monitored
The problem
17. @DevPaco
Code coverage
“The degree to which the source code of a program is
executed when a particular test suite runs”
Monitoring Tests
18. @DevPaco
Example
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
19. @DevPaco
Method coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
20. @DevPaco
Method coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
21. @DevPaco
Statement coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
22. @DevPaco
Statement coverage
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));
23. @DevPaco
Condition coverage
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
24. @DevPaco
Condition coverage
TESTS:
testX: submit(proposal, 0, Instant.now().plusSeconds(999));
testY: submit(proposal, 5, Instant.now().plusSeconds(999));
testZ: submit(proposal, 0, Instant.now().minusSeconds(999));
public Result submit(Proposal proposal, int openProposals, Instant deadline) {
if (openProposals >= 3 || Instant.now().isAfter(deadline)) {
notificationService.notifySubmitFailed(proposal.getUser());
return new Error("Not allowed to submit");
}
// Doing the actual submit
return new Success();
}
25. @DevPaco
Pros:
• Helps you write more/better tests
• Easy/cheap to measure
• Shows what you didn’t test
• Shows that what you did test, didn’t crash
Code coverage
27. @DevPaco
But:
• Can be misleading
• Doesn’t give any guarantees
Code coverage
@Test
public void shouldReturnSuccessOnValidSubmit() {
proposalService.submit(randomProposal(), 0, tomorrow());
}
28. @DevPaco
• Testing the tests
• “Bug detection ability”
Test effectiveness
https://shirt.woot.com/offers/bug-hunt
43. @DevPaco
• Reduce number of tests to run
<configuration>
<excludedTestClasses>foo.bar.acceptance.*</excludedTestClasses>
<excludedTestClasses>foo.bar.database.*</excludedTestClasses>
<excludedTestClasses>foo.bar.slooow.*</excludedTestClasses>
</configuration>
Tweaking performance
51. @DevPaco
• Add the maven goal to your build step
mvn org.pitest:pitest-maven:mutationCoverage
• Plugins available for Jenkins/Sonar
• Pitest can be configured to break the pipeline
Adding mutation testing to CI
53. @DevPaco
Are there lives at stake?
Are you building software for a rocket?
Is it a self driving car?
When to consider mutation testing?
54. @DevPaco
Are you using code coverage?
What is the cost of fixing a bug?
Does the team think it’s a good idea?
When to consider mutation testing?
55. @DevPaco
• Code coverage: “How much of the code is tested”
• Mutation testing: “How proper is the code tested”
• Start small
• Tweak for performance!
Summary
56. @DevPaco
Thank you!
https://pitest.org/
Mutation testing for Java, Kotlin(paid)
https://stryker-mutator.io/
Supports JavaScript (typescript), C#, Scala
https://github.com/boxed/mutmut
Mutation testing tool for Python
@DevPaco
For questions, feedback, suggestions