SlideShare uma empresa Scribd logo
1 de 56
Baixar para ler offline
PIT: state of the art of mutation testing system
MUTANTS KILLER
by Tàrin Gamberìni - CC BY-NC-SA 4.0
Lavora da sempre con tecnologie
Java-based free e Open Source
presso la Regione Emilia-Romagna
Si interessa di Object Oriented Design,
Design Patterns, Build Automation,
Testing e Continuous Integration
TÀRIN GAMBERÌNI
Partecipa attivamente
al Java User Group
Padova
Unit Testing
Test-Driven
Development (TDD)
Continuous Integration
Code Coverage
Mutation Testing
DEVELOPMENT
UNIT TESTING & TDD
The proof of working code
Refactor with confidence (regression)
Improve code design
Living documentation
TDD extremely valuable in some context
CONTINUOUS INTEGRATION
Maintain a single source repository
Automate the build
Cheap statics for code quality
Automate unit & integration tests
Catch problems fast
LINE COVERAGE
Is a measure of tested source code
line, statement, branch, … , coverage
LINE COVERAGE 100%
Only able to identify not tested code
Doesn't check tests are able to detect faults
Measures only which code is executed by tests
WEAK TESTS
public static String foo( boolean b ) {
if ( b ) {
performVitallyImportantBusinessFunction();
return "OK";
}
return "FAIL";
}
@Test
public void shouldFailWhenGivenFalse() {
assertEquals("FAIL", foo( false ));
}
@Test
public void shouldBeOkWhenGivenTrue() {
assertEquals("OK", foo( true ));
}
The untested side effect
WEAK TESTS
public static String foo( boolean b ) {
if ( b ) {
performVitallyImportantBusinessFunction();
return "OK";
}
return "FAIL";
}
@Test
public void shouldFailWhenGivenFalse() {
assertEquals("FAIL", foo( false ));
}
@Test
public void shouldBeOkWhenGivenTrue() {
assertEquals("OK", foo( true ));
}
The untested side effect
WEAK TESTS
public static String foo( int i ) {
if ( i >= 0 ) {
return "foo";
} else {
return "bar";
}
}
@Test
public void shouldReturnBarWhenGiven1() {
assertEquals("bar", foo( 1 ));
}
@Test
public void shouldReturnFooWhenGivenMinus1() {
assertEquals("foo", foo( -1 ));
}
The missing boundary test
WEAK TESTS
public static String foo( int i ) {
if ( i >= 0 ) {
return "foo";
} else {
return "bar";
}
}
@Test
public void shouldReturnBarWhenGiven1() {
assertEquals("bar", foo( 1 ));
}
@Test
public void shouldReturnFooWhenGivenZero() {
assertEquals("foo", foo( 0 ));
}
@Test
public void shouldReturnFooWhenGivenMinus1() {
assertEquals("foo", foo( -1 ));
}
The missing boundary test
WEAK TESTS
public static String foo(Collaborator c, boolean b) {
if ( b ) {
return c.performAction();
}
return "FOO";
} @Test
public void shouldPerformActionWhenGivenTrue() {
foo(mockCollaborator,true);
verify(mockCollaborator).performAction();
}
@Test
public void shouldNotPerformActionWhenGivenFalse() {
foo(mockCollaborator,false);
verify(never(),mockCollaborator).performAction();
}
The myopic mockist
WEAK TESTS
public static String foo(Collaborator c, boolean b) {
if ( b ) {
return c.performAction();
}
return "FOO";
} @Test
public void shouldPerformActionWhenGivenTrue() {
String result = foo(mockCollaborator,true);
verify(mockCollaborator).performAction();
assertEquals("MOCK", result);
}
@Test
public void shouldNotPerformActionWhenGivenFalse() {
String result = foo(mockCollaborator,false);
verify(never(),mockCollaborator).performAction();
assertEquals("FOO", result);
}
The myopic mockist
WHO TESTS THE TESTS ?
MUTATION TESTING
●
originally proposed by Richard Lipton as a
student in 1971
●
1st implementation tool was a PhD work
in 1980
●
recent availability of massive computing
power has led to a resurgence
●
expose weaknesses in tests (quality)
MUTATION TESTING
●
originally proposed by Richard Lipton as a
student in 1971
●
1st implementation tool was a PhD work
in 1980
●
recent availability of massive computing
power has led to a resurgence
●
expose weaknesses in tests (quality)
MUTATION
Is a small change in your program
MUTANT
A
mutated
version of
your
program
MUTATION TESTING
generate lots of
mutants
run all tests against all mutants
check mutants:
killed or lived?
MUTANT KILLED
Proof of detected mutated code
Test is testing source code properly
If the unit test fails
MUTANT LIVED
Proof of not detected mutated code
If the unit test succeed
Test isn't testing source code
PIT: BASIC CONCEPTS
Applies a configurable set of mutators
Generates reports with test results
Manipulates the byte code generated
by compiling your code
PIT: MUTATORS
Conditionals Boundary
< → <=
<= → <
> → >=
>= → >
PIT: MUTATORS
Negate Conditionals
== → !=
!= → ==
<= → >
>= → <
< → >=
> → <=
PIT: MUTATORS
Math Mutator
+ → -
- → +
* → /
/ → *
% → *
& → |
| → &
^ → &
<< → >>
>> → <<
>>> → <<
PIT: MUTATORS
Remove Conditionals
if (a == b) {
// do something
}
if (true) {
// do something
}
→
PIT: MUTATORS
Void Method Call Mutator
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void
doSomething(int i) {
// does something
}
→
public int foo() {
int i = 5;
return i;
}
public void
doSomething(int i) {
// does something
}
PIT: MUTATORS
Many others
Activated by default
Conditionals Boundary Mutator - Increments Mutator -
Invert Negatives Mutator - Math Mutator - Negate
Conditionals Mutator - Return Values Mutator - Void
Method Calls Mutator
Deactivated by default
Constructor Calls Mutator - Inline Constant Mutator - Non
Void Method Calls Mutator - Remove Conditionals Mutator
- Experimental Member Variable Mutator - Experimental
Switch Mutator
HOW PIT WORKS
Running the tests
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
Your program
HOW PIT WORKS
Running the tests
PIT runs traditional line coverage analysis
HOW PIT WORKS
Running the tests
Generates mutant by applaying mutators
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
mutant 2
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
mutant 1
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
mutant 3
HOW PIT WORKS
Brute force
x = 9
100 tests x 1.2ms = 0.12s
100 tests x 1.2ms x 10000 mutants = 20min !!!
1000 class x 10 mutants per class = 10000 mutants
HOW PIT WORKS
Running the tests
PIT leverages line coverage to discard tests
that do not exercise the mutated line of code
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
public int foo() {
int i = 5;
doSomething(i);
return i;
}
public void doSomething(int i) {
// does something
}
mutant 3
HOW PIT WORKS
Extremely fast
x = 1
x = 1
x = 2 4 < 9
HOW PIT WORKS
Outcomes
● Killed, Lived
● No coverage: the same as Lived except there were no
tests that exercised the mutated line of code
● Non viable: bytecode was in some way invalid
● Timed Out: infinite loop
● Memory error: mutation that increases the amount of
memory used
● Run error: a large number of run errors probably
means something went wrong
PIT ECOSYSTEM
Tools
Command line
Other IDEs as a
Java application
( )
EXAMPLE: LINUXDAY2015
class Ticket
EXAMPLE: LINUXDAY2015
testGetPrice_withAge11
EXAMPLE: LINUXDAY2015
testGetPrice_withAge71
EXAMPLE: LINUXDAY2015
testGetPrice_withAge18
EXAMPLE: LINUXDAY2015
mvn org.pitest:pitest-maven:mutationCoverage
>> Generated 8 mutations Killed 6 (75%)
>> Ran 11 tests (1.38 tests per mutation)
=================================================================
- Mutators
=================================================================
> org.pitest...ConditionalsBoundaryMutator
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 2 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
-----------------------------------------------------------------
> org.pitest...ReturnValsMutator
>> Generated 3 Killed 3 (100%)
> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
EXAMPLE: LINUXDAY2015
EXAMPLE: LINUXDAY2015
EXAMPLE: LINUXDAY2015
testGetPrice_withAge12
EXAMPLE: LINUXDAY2015
mvn org.pitest:pitest-maven:mutationCoverage
>> Generated 8 mutations Killed 7 (88%)
>> Ran 10 tests (1.25 tests per mutation)
=================================================================
- Mutators
=================================================================
> org.pitest...ConditionalsBoundaryMutator
>> Generated 2 Killed 1 (50%)
> KILLED 1 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
-----------------------------------------------------------------
> org.pitest...ReturnValsMutator
>> Generated 3 Killed 3 (100%)
> KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
EXAMPLE: LINUXDAY2015
EXAMPLE: LINUXDAY2015
EXAMPLE: LINUXDAY2015
testGetPrice_withAge70
EXAMPLE: LINUXDAY2015
?!?!?!?!?
EXAMPLE: LINUXDAY2015
testGetPrice_withAge70
EXAMPLE: LINUXDAY2015
source code on GitHub
https://github.com/taringamberini/linuxday2015.git
QUESTIONS ?
CONTACTS
Tàrin Gamberìni
tarin.gamberini@jugpadova.it
www.taringamberini.com
CREDITS
Minion with gun - by jpartwork (no license available)
https://nanookofthenerd.wordpress.com/2015/01/30/the-magnificent-maddening-marvelous-
minion-episode-032/
Minion Hands on - by Ceb1031 (CC BY-SA 3.0)
http://parody.wikia.com/wiki/File:Bob_minions_hands.jpg
Canon XTi components after disassembly - by particlem (CC BY 2.0)
https://www.flickr.com/photos/particlem/3860278043/
Continuous Integration - by New Media Labs (no license available)
http://newmedialabs.co.za/approach
Rally car - by mmphotography.it (CC BY-NC 2.0)
https://www.flickr.com/photos/grantuking/9705677549/
Crash Test dummies for sale - by Jack French (CC BY-NC 2.0)
https://www.flickr.com/photos/jackfrench/34523572/
Minion (no license available)
http://thisandthatforkidsblog.com/2013/09/17/easy-to-make-diy-minion-halloween-pumpkin/
Images
Mutant - by minions fans-d6txvph.jpg (no license available)
http://despicableme.wikia.com/wiki/File:Evil_minions_by_minions_fans-d6txvph.jpg
Mutant killed - by Patricia Uribe (no license available)
http://weheartit.com/entry/70334753
Mutant lived - by Carlos Hernández (no license available)
https://gifcept.com/VUQrTOr.gif
PIT logo - by Ling Yeung (CC BY-NC-SA 3.0)
http://pitest.org/about/
Circled Minions - by HARDEEP BHOGAL - (no license available)
https://plus.google.com/109649247879792393053/posts/ghH6tiuUZni?
pid=6004685782480998482&oid=109649247879792393053
Finger puppets – by Gary Leeming (CC BY-NC-SA 2.0)
https://www.flickr.com/photos/grazulis/4754781005/
Questions - by Shinichi Izumi (CC BY-SA)
http://despicableme.wikia.com/wiki/File:Purple_Minion.png
CREDITS
Images

Mais conteúdo relacionado

Mais procurados

Automatisiertes Testen von Software in C++ (mit dem Test Framework Google Test)
Automatisiertes Testen von Software in C++ (mit dem Test Framework Google Test)Automatisiertes Testen von Software in C++ (mit dem Test Framework Google Test)
Automatisiertes Testen von Software in C++ (mit dem Test Framework Google Test)Florian Wolters
 
Unit Test Lab - Why Write Unit Tests?
Unit Test Lab - Why Write Unit Tests?Unit Test Lab - Why Write Unit Tests?
Unit Test Lab - Why Write Unit Tests?Danny van Kasteel
 
Como criar e executar testes paralelos web usando Selenium e containers
Como criar e executar testes paralelos web usando Selenium e containersComo criar e executar testes paralelos web usando Selenium e containers
Como criar e executar testes paralelos web usando Selenium e containersElias Nogueira
 
Anomaly Detection for Real-World Systems
Anomaly Detection for Real-World SystemsAnomaly Detection for Real-World Systems
Anomaly Detection for Real-World SystemsManojit Nandi
 
Artificial intelligence in qa
Artificial intelligence in qaArtificial intelligence in qa
Artificial intelligence in qaTaras Lytvyn
 
Testing and Mocking Object - The Art of Mocking.
Testing and Mocking Object - The Art of Mocking.Testing and Mocking Object - The Art of Mocking.
Testing and Mocking Object - The Art of Mocking.Deepak Singhvi
 
AI Testing What Why and How To Do It?
AI Testing What Why and How To Do It?AI Testing What Why and How To Do It?
AI Testing What Why and How To Do It?KiwiQA
 
Basic interview questions for manual testing
Basic interview questions for manual testingBasic interview questions for manual testing
Basic interview questions for manual testingJYOTI RANJAN PAL
 
Test-Driven Development (TDD)
Test-Driven Development (TDD)Test-Driven Development (TDD)
Test-Driven Development (TDD)Brian Rasmussen
 
Saving Time By Testing With Jest
Saving Time By Testing With JestSaving Time By Testing With Jest
Saving Time By Testing With JestBen McCormick
 
05 junit
05 junit05 junit
05 junitmha4
 
C++ Unit Test with Google Testing Framework
C++ Unit Test with Google Testing FrameworkC++ Unit Test with Google Testing Framework
C++ Unit Test with Google Testing FrameworkHumberto Marchezi
 
Moq presentation
Moq presentationMoq presentation
Moq presentationLynxStar
 
Test Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.comTest Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.comIdexcel Technologies
 
10 Fragen vor Testautomatisierung
10 Fragen vor Testautomatisierung10 Fragen vor Testautomatisierung
10 Fragen vor TestautomatisierungClaudia Baur
 
Integration Group - Robot Framework
Integration Group - Robot Framework Integration Group - Robot Framework
Integration Group - Robot Framework OpenDaylight
 
자동화된 Test Case의 효과
자동화된 Test Case의 효과자동화된 Test Case의 효과
자동화된 Test Case의 효과도형 임
 

Mais procurados (20)

Automatisiertes Testen von Software in C++ (mit dem Test Framework Google Test)
Automatisiertes Testen von Software in C++ (mit dem Test Framework Google Test)Automatisiertes Testen von Software in C++ (mit dem Test Framework Google Test)
Automatisiertes Testen von Software in C++ (mit dem Test Framework Google Test)
 
Introdução ao design de teste de software
Introdução ao design de teste de softwareIntrodução ao design de teste de software
Introdução ao design de teste de software
 
Unit Test Lab - Why Write Unit Tests?
Unit Test Lab - Why Write Unit Tests?Unit Test Lab - Why Write Unit Tests?
Unit Test Lab - Why Write Unit Tests?
 
Como criar e executar testes paralelos web usando Selenium e containers
Como criar e executar testes paralelos web usando Selenium e containersComo criar e executar testes paralelos web usando Selenium e containers
Como criar e executar testes paralelos web usando Selenium e containers
 
Anomaly Detection for Real-World Systems
Anomaly Detection for Real-World SystemsAnomaly Detection for Real-World Systems
Anomaly Detection for Real-World Systems
 
Artificial intelligence in qa
Artificial intelligence in qaArtificial intelligence in qa
Artificial intelligence in qa
 
Testing and Mocking Object - The Art of Mocking.
Testing and Mocking Object - The Art of Mocking.Testing and Mocking Object - The Art of Mocking.
Testing and Mocking Object - The Art of Mocking.
 
AI Testing What Why and How To Do It?
AI Testing What Why and How To Do It?AI Testing What Why and How To Do It?
AI Testing What Why and How To Do It?
 
Basic interview questions for manual testing
Basic interview questions for manual testingBasic interview questions for manual testing
Basic interview questions for manual testing
 
Uft Basics
Uft BasicsUft Basics
Uft Basics
 
Test-Driven Development (TDD)
Test-Driven Development (TDD)Test-Driven Development (TDD)
Test-Driven Development (TDD)
 
Saving Time By Testing With Jest
Saving Time By Testing With JestSaving Time By Testing With Jest
Saving Time By Testing With Jest
 
05 junit
05 junit05 junit
05 junit
 
Unit testing with java
Unit testing with javaUnit testing with java
Unit testing with java
 
C++ Unit Test with Google Testing Framework
C++ Unit Test with Google Testing FrameworkC++ Unit Test with Google Testing Framework
C++ Unit Test with Google Testing Framework
 
Moq presentation
Moq presentationMoq presentation
Moq presentation
 
Test Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.comTest Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.com
 
10 Fragen vor Testautomatisierung
10 Fragen vor Testautomatisierung10 Fragen vor Testautomatisierung
10 Fragen vor Testautomatisierung
 
Integration Group - Robot Framework
Integration Group - Robot Framework Integration Group - Robot Framework
Integration Group - Robot Framework
 
자동화된 Test Case의 효과
자동화된 Test Case의 효과자동화된 Test Case의 효과
자동화된 Test Case의 효과
 

Destaque

MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system Tarin Gamberini
 
Commit messages - Good practices
Commit messages - Good practicesCommit messages - Good practices
Commit messages - Good practicesTarin Gamberini
 
Mutation Testing with PIT (Booster 2014, 2014-MAR-13)
Mutation Testing with PIT (Booster 2014, 2014-MAR-13)Mutation Testing with PIT (Booster 2014, 2014-MAR-13)
Mutation Testing with PIT (Booster 2014, 2014-MAR-13)Filip Van Laenen
 
أخطاء برمجية | programming errors
أخطاء برمجية | programming errorsأخطاء برمجية | programming errors
أخطاء برمجية | programming errorsnawal saad
 
Igor Popov: Mutation Testing at I T.A.K.E. Unconference 2015
Igor Popov: Mutation Testing at I T.A.K.E. Unconference 2015Igor Popov: Mutation Testing at I T.A.K.E. Unconference 2015
Igor Popov: Mutation Testing at I T.A.K.E. Unconference 2015Mozaic Works
 
Mutation Testing
Mutation TestingMutation Testing
Mutation TestingESUG
 
Resources for veterans and career professionals
Resources for veterans and career professionalsResources for veterans and career professionals
Resources for veterans and career professionalsAmy Armstrong
 
JS-resume final
JS-resume finalJS-resume final
JS-resume finalJuan Silva
 
Entertainment on the pier
Entertainment on the pierEntertainment on the pier
Entertainment on the pier1066heritage
 
XMPP Hands-On
XMPP Hands-OnXMPP Hands-On
XMPP Hands-Oncodebits
 
Boletin aprender a aprender
Boletin aprender a aprenderBoletin aprender a aprender
Boletin aprender a aprenderLUISA1LEON
 
Máximos Goleadores de la historia
Máximos Goleadores de la historiaMáximos Goleadores de la historia
Máximos Goleadores de la historiajaimejcidead
 
фольклорні балади про робін гуда
фольклорні балади про робін гудафольклорні балади про робін гуда
фольклорні балади про робін гудаSnezhana Pshenichnaya
 
Entrevista a Mia Couto
Entrevista a Mia CoutoEntrevista a Mia Couto
Entrevista a Mia CoutoSandra Branco
 
Understanding Open Licensing: Day Three - Progressing Towards Opennes
Understanding Open Licensing: Day Three - Progressing Towards OpennesUnderstanding Open Licensing: Day Three - Progressing Towards Opennes
Understanding Open Licensing: Day Three - Progressing Towards OpennesIDS Knowledge Services
 

Destaque (20)

MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
MUTANTS KILLER (Revised) - PIT: state of the art of mutation testing system
 
Commit messages - Good practices
Commit messages - Good practicesCommit messages - Good practices
Commit messages - Good practices
 
Apache Maven
Apache MavenApache Maven
Apache Maven
 
Mutation Testing with PIT (Booster 2014, 2014-MAR-13)
Mutation Testing with PIT (Booster 2014, 2014-MAR-13)Mutation Testing with PIT (Booster 2014, 2014-MAR-13)
Mutation Testing with PIT (Booster 2014, 2014-MAR-13)
 
أخطاء برمجية | programming errors
أخطاء برمجية | programming errorsأخطاء برمجية | programming errors
أخطاء برمجية | programming errors
 
Igor Popov: Mutation Testing at I T.A.K.E. Unconference 2015
Igor Popov: Mutation Testing at I T.A.K.E. Unconference 2015Igor Popov: Mutation Testing at I T.A.K.E. Unconference 2015
Igor Popov: Mutation Testing at I T.A.K.E. Unconference 2015
 
Mutation testing
Mutation testingMutation testing
Mutation testing
 
Mutation testing in Java
Mutation testing in JavaMutation testing in Java
Mutation testing in Java
 
Mutation Testing
Mutation TestingMutation Testing
Mutation Testing
 
تحليل النظم
تحليل النظمتحليل النظم
تحليل النظم
 
Resources for veterans and career professionals
Resources for veterans and career professionalsResources for veterans and career professionals
Resources for veterans and career professionals
 
JS-resume final
JS-resume finalJS-resume final
JS-resume final
 
Entertainment on the pier
Entertainment on the pierEntertainment on the pier
Entertainment on the pier
 
XMPP Hands-On
XMPP Hands-OnXMPP Hands-On
XMPP Hands-On
 
Promising Practices for Students Transitioning to Independent Schools
Promising Practices for Students Transitioning to Independent SchoolsPromising Practices for Students Transitioning to Independent Schools
Promising Practices for Students Transitioning to Independent Schools
 
Boletin aprender a aprender
Boletin aprender a aprenderBoletin aprender a aprender
Boletin aprender a aprender
 
Máximos Goleadores de la historia
Máximos Goleadores de la historiaMáximos Goleadores de la historia
Máximos Goleadores de la historia
 
фольклорні балади про робін гуда
фольклорні балади про робін гудафольклорні балади про робін гуда
фольклорні балади про робін гуда
 
Entrevista a Mia Couto
Entrevista a Mia CoutoEntrevista a Mia Couto
Entrevista a Mia Couto
 
Understanding Open Licensing: Day Three - Progressing Towards Opennes
Understanding Open Licensing: Day Three - Progressing Towards OpennesUnderstanding Open Licensing: Day Three - Progressing Towards Opennes
Understanding Open Licensing: Day Three - Progressing Towards Opennes
 

Semelhante a MUTANTS KILLER - PIT: state of the art of mutation testing system

Kill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van RijnKill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van RijnNLJUG
 
Must.Kill.Mutants. Agile Testing Days 2017
Must.Kill.Mutants. Agile Testing Days 2017Must.Kill.Mutants. Agile Testing Days 2017
Must.Kill.Mutants. Agile Testing Days 2017Gerald Muecke
 
Must.kill.mutants. TopConf Tallinn 2016
Must.kill.mutants. TopConf Tallinn 2016Must.kill.mutants. TopConf Tallinn 2016
Must.kill.mutants. TopConf Tallinn 2016Gerald Muecke
 
Test Driven Development with JavaFX
Test Driven Development with JavaFXTest Driven Development with JavaFX
Test Driven Development with JavaFXHendrik Ebbers
 
Никита Галкин "Testing in Frontend World"
Никита Галкин "Testing in Frontend World"Никита Галкин "Testing in Frontend World"
Никита Галкин "Testing in Frontend World"Fwdays
 
Unit Testing with JUnit4 by Ravikiran Janardhana
Unit Testing with JUnit4 by Ravikiran JanardhanaUnit Testing with JUnit4 by Ravikiran Janardhana
Unit Testing with JUnit4 by Ravikiran JanardhanaRavikiran J
 
Testes? Mas isso não aumenta o tempo de projecto? Não quero...
Testes? Mas isso não aumenta o tempo de projecto? Não quero...Testes? Mas isso não aumenta o tempo de projecto? Não quero...
Testes? Mas isso não aumenta o tempo de projecto? Não quero...Comunidade NetPonto
 
31b - JUnit and Mockito.pdf
31b - JUnit and Mockito.pdf31b - JUnit and Mockito.pdf
31b - JUnit and Mockito.pdfgauravavam
 
Do I need tests when I have the compiler - Andrzej Jóźwiak - TomTom Dev Day 2020
Do I need tests when I have the compiler - Andrzej Jóźwiak - TomTom Dev Day 2020Do I need tests when I have the compiler - Andrzej Jóźwiak - TomTom Dev Day 2020
Do I need tests when I have the compiler - Andrzej Jóźwiak - TomTom Dev Day 2020Andrzej Jóźwiak
 
Stopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestStopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestSeb Rose
 
DSR Testing (Part 1)
DSR Testing (Part 1)DSR Testing (Part 1)
DSR Testing (Part 1)Steve Upton
 
Java Unit Testing Tool Competition — Fifth Round
Java Unit Testing Tool Competition — Fifth RoundJava Unit Testing Tool Competition — Fifth Round
Java Unit Testing Tool Competition — Fifth RoundAnnibale Panichella
 
Don't Be Mocked by your Mocks - Best Practices using Mocks
Don't Be Mocked by your Mocks - Best Practices using MocksDon't Be Mocked by your Mocks - Best Practices using Mocks
Don't Be Mocked by your Mocks - Best Practices using MocksVictor Rentea
 
Wix Automation - The False Positive Paradox
Wix Automation - The False Positive ParadoxWix Automation - The False Positive Paradox
Wix Automation - The False Positive ParadoxEfrat Attas
 
GlobalLogic Test Automation Online TechTalk “Test Driven Development as a Per...
GlobalLogic Test Automation Online TechTalk “Test Driven Development as a Per...GlobalLogic Test Automation Online TechTalk “Test Driven Development as a Per...
GlobalLogic Test Automation Online TechTalk “Test Driven Development as a Per...GlobalLogic Ukraine
 
TDD And Refactoring
TDD And RefactoringTDD And Refactoring
TDD And RefactoringNaresh Jain
 
Hitchhiker's guide to Functional Testing
Hitchhiker's guide to Functional TestingHitchhiker's guide to Functional Testing
Hitchhiker's guide to Functional TestingWiebe Elsinga
 

Semelhante a MUTANTS KILLER - PIT: state of the art of mutation testing system (20)

Test driven development
Test driven developmentTest driven development
Test driven development
 
Kill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van RijnKill the mutants and test your tests - Roy van Rijn
Kill the mutants and test your tests - Roy van Rijn
 
Must.Kill.Mutants. Agile Testing Days 2017
Must.Kill.Mutants. Agile Testing Days 2017Must.Kill.Mutants. Agile Testing Days 2017
Must.Kill.Mutants. Agile Testing Days 2017
 
Must.kill.mutants. TopConf Tallinn 2016
Must.kill.mutants. TopConf Tallinn 2016Must.kill.mutants. TopConf Tallinn 2016
Must.kill.mutants. TopConf Tallinn 2016
 
Unit testing - A&BP CC
Unit testing - A&BP CCUnit testing - A&BP CC
Unit testing - A&BP CC
 
Test Driven Development with JavaFX
Test Driven Development with JavaFXTest Driven Development with JavaFX
Test Driven Development with JavaFX
 
Никита Галкин "Testing in Frontend World"
Никита Галкин "Testing in Frontend World"Никита Галкин "Testing in Frontend World"
Никита Галкин "Testing in Frontend World"
 
Unit Testing with JUnit4 by Ravikiran Janardhana
Unit Testing with JUnit4 by Ravikiran JanardhanaUnit Testing with JUnit4 by Ravikiran Janardhana
Unit Testing with JUnit4 by Ravikiran Janardhana
 
Testes? Mas isso não aumenta o tempo de projecto? Não quero...
Testes? Mas isso não aumenta o tempo de projecto? Não quero...Testes? Mas isso não aumenta o tempo de projecto? Não quero...
Testes? Mas isso não aumenta o tempo de projecto? Não quero...
 
31b - JUnit and Mockito.pdf
31b - JUnit and Mockito.pdf31b - JUnit and Mockito.pdf
31b - JUnit and Mockito.pdf
 
Do I need tests when I have the compiler - Andrzej Jóźwiak - TomTom Dev Day 2020
Do I need tests when I have the compiler - Andrzej Jóźwiak - TomTom Dev Day 2020Do I need tests when I have the compiler - Andrzej Jóźwiak - TomTom Dev Day 2020
Do I need tests when I have the compiler - Andrzej Jóźwiak - TomTom Dev Day 2020
 
Advanced Java Testing
Advanced Java TestingAdvanced Java Testing
Advanced Java Testing
 
Stopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under TestStopping the Rot - Putting Legacy C++ Under Test
Stopping the Rot - Putting Legacy C++ Under Test
 
DSR Testing (Part 1)
DSR Testing (Part 1)DSR Testing (Part 1)
DSR Testing (Part 1)
 
Java Unit Testing Tool Competition — Fifth Round
Java Unit Testing Tool Competition — Fifth RoundJava Unit Testing Tool Competition — Fifth Round
Java Unit Testing Tool Competition — Fifth Round
 
Don't Be Mocked by your Mocks - Best Practices using Mocks
Don't Be Mocked by your Mocks - Best Practices using MocksDon't Be Mocked by your Mocks - Best Practices using Mocks
Don't Be Mocked by your Mocks - Best Practices using Mocks
 
Wix Automation - The False Positive Paradox
Wix Automation - The False Positive ParadoxWix Automation - The False Positive Paradox
Wix Automation - The False Positive Paradox
 
GlobalLogic Test Automation Online TechTalk “Test Driven Development as a Per...
GlobalLogic Test Automation Online TechTalk “Test Driven Development as a Per...GlobalLogic Test Automation Online TechTalk “Test Driven Development as a Per...
GlobalLogic Test Automation Online TechTalk “Test Driven Development as a Per...
 
TDD And Refactoring
TDD And RefactoringTDD And Refactoring
TDD And Refactoring
 
Hitchhiker's guide to Functional Testing
Hitchhiker's guide to Functional TestingHitchhiker's guide to Functional Testing
Hitchhiker's guide to Functional Testing
 

Último

Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension AidPhilip Schwarz
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...ICS
 
10 Trends Likely to Shape Enterprise Technology in 2024
10 Trends Likely to Shape Enterprise Technology in 202410 Trends Likely to Shape Enterprise Technology in 2024
10 Trends Likely to Shape Enterprise Technology in 2024Mind IT Systems
 
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerHow To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerThousandEyes
 
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdfintroduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdfVishalKumarJha10
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisamasabamasaba
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️Delhi Call girls
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...Jittipong Loespradit
 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providermohitmore19
 
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionIntroducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionOnePlan Solutions
 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Modelsaagamshah0812
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnAmarnathKambale
 
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...Shane Coughlan
 
Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf
Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdfAzure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf
Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdfryanfarris8
 
LEVEL 5 - SESSION 1 2023 (1).pptx - PDF 123456
LEVEL 5   - SESSION 1 2023 (1).pptx - PDF 123456LEVEL 5   - SESSION 1 2023 (1).pptx - PDF 123456
LEVEL 5 - SESSION 1 2023 (1).pptx - PDF 123456KiaraTiradoMicha
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfkalichargn70th171
 
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Steffen Staab
 
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsRight Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsJhone kinadey
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsArshad QA
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...Health
 

Último (20)

Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
Direct Style Effect Systems -The Print[A] Example- A Comprehension AidDirect Style Effect Systems -The Print[A] Example- A Comprehension Aid
Direct Style Effect Systems - The Print[A] Example - A Comprehension Aid
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
 
10 Trends Likely to Shape Enterprise Technology in 2024
10 Trends Likely to Shape Enterprise Technology in 202410 Trends Likely to Shape Enterprise Technology in 2024
10 Trends Likely to Shape Enterprise Technology in 2024
 
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected WorkerHow To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
 
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdfintroduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
 
TECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service providerTECUNIQUE: Success Stories: IT Service provider
TECUNIQUE: Success Stories: IT Service provider
 
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) SolutionIntroducing Microsoft’s new Enterprise Work Management (EWM) Solution
Introducing Microsoft’s new Enterprise Work Management (EWM) Solution
 
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language ModelsUnlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Models
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
 
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
OpenChain - The Ramifications of ISO/IEC 5230 and ISO/IEC 18974 for Legal Pro...
 
Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf
Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdfAzure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf
Azure_Native_Qumulo_High_Performance_Compute_Benchmarks.pdf
 
LEVEL 5 - SESSION 1 2023 (1).pptx - PDF 123456
LEVEL 5   - SESSION 1 2023 (1).pptx - PDF 123456LEVEL 5   - SESSION 1 2023 (1).pptx - PDF 123456
LEVEL 5 - SESSION 1 2023 (1).pptx - PDF 123456
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
 
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...
 
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial GoalsRight Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
 
Software Quality Assurance Interview Questions
Software Quality Assurance Interview QuestionsSoftware Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 

MUTANTS KILLER - PIT: state of the art of mutation testing system

  • 1. PIT: state of the art of mutation testing system MUTANTS KILLER by Tàrin Gamberìni - CC BY-NC-SA 4.0
  • 2. Lavora da sempre con tecnologie Java-based free e Open Source presso la Regione Emilia-Romagna Si interessa di Object Oriented Design, Design Patterns, Build Automation, Testing e Continuous Integration TÀRIN GAMBERÌNI Partecipa attivamente al Java User Group Padova
  • 3. Unit Testing Test-Driven Development (TDD) Continuous Integration Code Coverage Mutation Testing DEVELOPMENT
  • 4. UNIT TESTING & TDD The proof of working code Refactor with confidence (regression) Improve code design Living documentation TDD extremely valuable in some context
  • 5. CONTINUOUS INTEGRATION Maintain a single source repository Automate the build Cheap statics for code quality Automate unit & integration tests Catch problems fast
  • 6. LINE COVERAGE Is a measure of tested source code line, statement, branch, … , coverage
  • 7. LINE COVERAGE 100% Only able to identify not tested code Doesn't check tests are able to detect faults Measures only which code is executed by tests
  • 8. WEAK TESTS public static String foo( boolean b ) { if ( b ) { performVitallyImportantBusinessFunction(); return "OK"; } return "FAIL"; } @Test public void shouldFailWhenGivenFalse() { assertEquals("FAIL", foo( false )); } @Test public void shouldBeOkWhenGivenTrue() { assertEquals("OK", foo( true )); } The untested side effect
  • 9. WEAK TESTS public static String foo( boolean b ) { if ( b ) { performVitallyImportantBusinessFunction(); return "OK"; } return "FAIL"; } @Test public void shouldFailWhenGivenFalse() { assertEquals("FAIL", foo( false )); } @Test public void shouldBeOkWhenGivenTrue() { assertEquals("OK", foo( true )); } The untested side effect
  • 10. WEAK TESTS public static String foo( int i ) { if ( i >= 0 ) { return "foo"; } else { return "bar"; } } @Test public void shouldReturnBarWhenGiven1() { assertEquals("bar", foo( 1 )); } @Test public void shouldReturnFooWhenGivenMinus1() { assertEquals("foo", foo( -1 )); } The missing boundary test
  • 11. WEAK TESTS public static String foo( int i ) { if ( i >= 0 ) { return "foo"; } else { return "bar"; } } @Test public void shouldReturnBarWhenGiven1() { assertEquals("bar", foo( 1 )); } @Test public void shouldReturnFooWhenGivenZero() { assertEquals("foo", foo( 0 )); } @Test public void shouldReturnFooWhenGivenMinus1() { assertEquals("foo", foo( -1 )); } The missing boundary test
  • 12. WEAK TESTS public static String foo(Collaborator c, boolean b) { if ( b ) { return c.performAction(); } return "FOO"; } @Test public void shouldPerformActionWhenGivenTrue() { foo(mockCollaborator,true); verify(mockCollaborator).performAction(); } @Test public void shouldNotPerformActionWhenGivenFalse() { foo(mockCollaborator,false); verify(never(),mockCollaborator).performAction(); } The myopic mockist
  • 13. WEAK TESTS public static String foo(Collaborator c, boolean b) { if ( b ) { return c.performAction(); } return "FOO"; } @Test public void shouldPerformActionWhenGivenTrue() { String result = foo(mockCollaborator,true); verify(mockCollaborator).performAction(); assertEquals("MOCK", result); } @Test public void shouldNotPerformActionWhenGivenFalse() { String result = foo(mockCollaborator,false); verify(never(),mockCollaborator).performAction(); assertEquals("FOO", result); } The myopic mockist
  • 14. WHO TESTS THE TESTS ?
  • 15. MUTATION TESTING ● originally proposed by Richard Lipton as a student in 1971 ● 1st implementation tool was a PhD work in 1980 ● recent availability of massive computing power has led to a resurgence ● expose weaknesses in tests (quality)
  • 16. MUTATION TESTING ● originally proposed by Richard Lipton as a student in 1971 ● 1st implementation tool was a PhD work in 1980 ● recent availability of massive computing power has led to a resurgence ● expose weaknesses in tests (quality)
  • 17. MUTATION Is a small change in your program
  • 19. MUTATION TESTING generate lots of mutants run all tests against all mutants check mutants: killed or lived?
  • 20. MUTANT KILLED Proof of detected mutated code Test is testing source code properly If the unit test fails
  • 21. MUTANT LIVED Proof of not detected mutated code If the unit test succeed Test isn't testing source code
  • 22.
  • 23. PIT: BASIC CONCEPTS Applies a configurable set of mutators Generates reports with test results Manipulates the byte code generated by compiling your code
  • 24. PIT: MUTATORS Conditionals Boundary < → <= <= → < > → >= >= → >
  • 25. PIT: MUTATORS Negate Conditionals == → != != → == <= → > >= → < < → >= > → <=
  • 26. PIT: MUTATORS Math Mutator + → - - → + * → / / → * % → * & → | | → & ^ → & << → >> >> → << >>> → <<
  • 27. PIT: MUTATORS Remove Conditionals if (a == b) { // do something } if (true) { // do something } →
  • 28. PIT: MUTATORS Void Method Call Mutator public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } → public int foo() { int i = 5; return i; } public void doSomething(int i) { // does something }
  • 29. PIT: MUTATORS Many others Activated by default Conditionals Boundary Mutator - Increments Mutator - Invert Negatives Mutator - Math Mutator - Negate Conditionals Mutator - Return Values Mutator - Void Method Calls Mutator Deactivated by default Constructor Calls Mutator - Inline Constant Mutator - Non Void Method Calls Mutator - Remove Conditionals Mutator - Experimental Member Variable Mutator - Experimental Switch Mutator
  • 30. HOW PIT WORKS Running the tests public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } Your program
  • 31. HOW PIT WORKS Running the tests PIT runs traditional line coverage analysis
  • 32. HOW PIT WORKS Running the tests Generates mutant by applaying mutators public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } mutant 2 public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } mutant 1 public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } mutant 3
  • 33. HOW PIT WORKS Brute force x = 9 100 tests x 1.2ms = 0.12s 100 tests x 1.2ms x 10000 mutants = 20min !!! 1000 class x 10 mutants per class = 10000 mutants
  • 34. HOW PIT WORKS Running the tests PIT leverages line coverage to discard tests that do not exercise the mutated line of code public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } public int foo() { int i = 5; doSomething(i); return i; } public void doSomething(int i) { // does something } mutant 3
  • 35. HOW PIT WORKS Extremely fast x = 1 x = 1 x = 2 4 < 9
  • 36. HOW PIT WORKS Outcomes ● Killed, Lived ● No coverage: the same as Lived except there were no tests that exercised the mutated line of code ● Non viable: bytecode was in some way invalid ● Timed Out: infinite loop ● Memory error: mutation that increases the amount of memory used ● Run error: a large number of run errors probably means something went wrong
  • 37. PIT ECOSYSTEM Tools Command line Other IDEs as a Java application ( )
  • 42. EXAMPLE: LINUXDAY2015 mvn org.pitest:pitest-maven:mutationCoverage >> Generated 8 mutations Killed 6 (75%) >> Ran 11 tests (1.38 tests per mutation) ================================================================= - Mutators ================================================================= > org.pitest...ConditionalsBoundaryMutator >> Generated 2 Killed 0 (0%) > KILLED 0 SURVIVED 2 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 ----------------------------------------------------------------- > org.pitest...ReturnValsMutator >> Generated 3 Killed 3 (100%) > KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
  • 46. EXAMPLE: LINUXDAY2015 mvn org.pitest:pitest-maven:mutationCoverage >> Generated 8 mutations Killed 7 (88%) >> Ran 10 tests (1.25 tests per mutation) ================================================================= - Mutators ================================================================= > org.pitest...ConditionalsBoundaryMutator >> Generated 2 Killed 1 (50%) > KILLED 1 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0 ----------------------------------------------------------------- > org.pitest...ReturnValsMutator >> Generated 3 Killed 3 (100%) > KILLED 3 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
  • 52. EXAMPLE: LINUXDAY2015 source code on GitHub https://github.com/taringamberini/linuxday2015.git
  • 55. CREDITS Minion with gun - by jpartwork (no license available) https://nanookofthenerd.wordpress.com/2015/01/30/the-magnificent-maddening-marvelous- minion-episode-032/ Minion Hands on - by Ceb1031 (CC BY-SA 3.0) http://parody.wikia.com/wiki/File:Bob_minions_hands.jpg Canon XTi components after disassembly - by particlem (CC BY 2.0) https://www.flickr.com/photos/particlem/3860278043/ Continuous Integration - by New Media Labs (no license available) http://newmedialabs.co.za/approach Rally car - by mmphotography.it (CC BY-NC 2.0) https://www.flickr.com/photos/grantuking/9705677549/ Crash Test dummies for sale - by Jack French (CC BY-NC 2.0) https://www.flickr.com/photos/jackfrench/34523572/ Minion (no license available) http://thisandthatforkidsblog.com/2013/09/17/easy-to-make-diy-minion-halloween-pumpkin/ Images
  • 56. Mutant - by minions fans-d6txvph.jpg (no license available) http://despicableme.wikia.com/wiki/File:Evil_minions_by_minions_fans-d6txvph.jpg Mutant killed - by Patricia Uribe (no license available) http://weheartit.com/entry/70334753 Mutant lived - by Carlos Hernández (no license available) https://gifcept.com/VUQrTOr.gif PIT logo - by Ling Yeung (CC BY-NC-SA 3.0) http://pitest.org/about/ Circled Minions - by HARDEEP BHOGAL - (no license available) https://plus.google.com/109649247879792393053/posts/ghH6tiuUZni? pid=6004685782480998482&oid=109649247879792393053 Finger puppets – by Gary Leeming (CC BY-NC-SA 2.0) https://www.flickr.com/photos/grazulis/4754781005/ Questions - by Shinichi Izumi (CC BY-SA) http://despicableme.wikia.com/wiki/File:Purple_Minion.png CREDITS Images