DSpot is a tool that Automatically detect and generate missing assertions for Junit test cases (also known as test amplification). For more information, see https://github.com/STAMP-project/dspot

1. Test amplification tutorial with DSpot
Benjamin DANGLOT
December 14th
, 2017
benjamin.danglot@inria.fr
STAMP Workshop Atos - Madrid
1/21

2. Use Case Example
2/21

3. DSpot

4. DSpot: Principle
Program
Test suite
3/21

5. DSpot: Principle
×
Program
Test suite Test criterion
3/21

6. DSpot: Principle
DSpot
×
Program
Test suite Test criterion
3/21

7. DSpot: Principle
DSpot
×
Program
Test suite Test criterion
Amplified
test suite
3/21

8. DSpot: Principle
DSpot
Program
Test suite Test criterion
Amplified
test suite
3/21

9. Use Case Example
4/21

10. Test Example
5/21

11. Test Example
In blue, the input of the test
5/21

12. Test Example
In blue, the input of the test
In green, an oracle of the test to verify the current behavior
5/21

13. How DSpot works?
6/21

14. DSpot: 1. Input Modification
6/21

15. DSpot: 1. Input Modification
Modifies the input to create new state of the program
7/21

16. DSpot: 1. Input Modification
Removes a method call to create a new state of the program
Remove method
call
7/21

17. DSpot: 1. Input Modification
Removes a method call to create a new state of the program
7/21

18. DSpot: 1. Input Modification
Removes a method call to create a new state of the program
Before:
benjamin.isHungry() → false
After:
benjamin.isHungry() → true
7/21

19. DSpot: 2. Assertion Generation
8/21

20. DSpot: 2. Assertion Generation
Goal: Generate assertions to verify the state of the program
1. Remove existing assertions
2. Instrument the test
3. Run Instrumented test to collect values
4. Generate assertions based on values collected
9/21

21. DSpot: 2. Assertion Generation
A. Removes existing assertions
10/21

22. DSpot: 2. Assertion Generation
assertion
removed
A. Removes existing assertions
11/21

23. DSpot: 2. Assertion Generation
B. Instruments the code to gather new values
12/21

24. DSpot: 2. Assertion Generation
B. Instruments the code to gather new values
13/21

25. DSpot: 2. Assertion Generation
C. Runs the instrumented test and obtains observations
14/21

26. DSpot: 2. Assertion Generation
C. Runs the instrumented test and obtains observations
Observations:
benjamin.isHungry() → true
benjamin.isHappy() → false
15/21

27. DSpot: 2. Assertion Generation
D. Generates new assertions based on observations
Observations:
benjamin.isHungry() → true
benjamin.isHappy() → false
16/21

28. DSpot: 2. Assertion Generation
D. Generates new assertions based on observations
Observations:
benjamin.isHungry() → true
benjamin.isHappy() → false
17/21

29. DSpot: Amplification of Test Results
Original test
Amplified test
18/21

30. DSpot: Amplification of Test Results
Original test
Method call removed
Assertion
Generated
Amplified test
18/21

31. DSpot: 3. Test Selection
19/21

32. Pre-Requisite: Mutation Analysis
Test Suite
Program
The mutation analysis computes the mutation score, a measure of the
efficiency of tests
20/21

33. Mutation Analysis
Test Suite
Program
Variant 1
Test Suite
Program
Variant 1
Inject Fault
First, creates variants of program by injecting faults
20/21

34. Mutation Analysis
Test Suite
Program
Variant 1
Variant 2
Test Suite
Program
Variant 1
Variant 2Inject Fault
Inject Fault
First, creates variants of program by injecting faults
20/21

35. Mutation Analysis
Test Suite
Program
Variant 1
Variant 2
Variant 3
Test Suite
Program
Variant 1
Variant 2
Variant 3
Inject Fault
Inject Fault
Inject Fault
First, creates variants of program by injecting faults
20/21

36. Mutation Analysis
Test Suite
Program
Variant 1
Variant 2
Variant 3Run
Run
Run
Then, runs tests against each variant
20/21

37. Mutation Analysis
Test Suite
Program
Variant 1
Variant 2
Variant 3Run
Run
Run
Killed
Killed
Alive
Failing test ⇔ mutant killed ⇔ behavior variation detected
20/21

38. Selection Of Test Using Mutation Analysis
21/21

39. Selection Of Test Using Mutation Analysis
21/21

40. Selection Of Test Using Mutation Analysis
21/21

41. Selection Of Test Using Mutation Analysis
21/21

42. Selection Of Test Using Mutation Analysis
21/21

43. Selection Of Test Using Mutation Analysis
21/21

44. Selection Of Test Using Mutation Analysis
21/21

45. Selection Of Test Using Mutation Analysis
21/21

46. DSpot: iteration
22/21

47. DSpot: Try it
https://github.com/danglotb/tavern
23/21

48. DSpot: Command Line Options
https://github.com/danglotb/tavern
• --path-to-properties
src/main/resources/tavern.properties: general description of
the project
• --iteration 2: number of time that the main loop is applied
• --test fr.inria.stamp.MainTest: full qualified name of the
test class to be amplified
• --cases test: name of the test method (test case)
• --amplifiers TestDataMutator: used input modification
• --test-criterion PitMutantScoreSelector: test criterion
selector used
• --verbose: enable more logs
• --path-pit-result mutations results/original.csv: read
mutation analysis from file to speed up the exe
24/21

49. Input Modification: Available Operators
• Literal Modification
1. Add/Remove/Replace a char in String
2. replace by Random String
3. +1/−1/×2/÷2 values
4. replace by random/upper bound/lower bound value
5. negate boolean
• Method Call Duplication
• Method Call Deletion
• Generation of Method Call (and required objects)
25/21

50. Test Selection: Available Selector
• Mutation Score
• Branch Coverage
• Instruction Coverage and Execution Path
• Behavior Changes
26/21

51. DSpot: Playing With Command Line Options
https://github.com/danglotb/tavern
Input Modification
--amplifiers:
• MethodAdd: duplicate
method call
• MethodRemove: remove
method call
• TestDataMutator: mutate
literals
• StatementAdd: add
method call
Compose them using ’:’
Test Criterion Selector
--test-criterion:
• PitMutantScoreSelector:
increase mutation score
• JacocoCoverageSelector:
increase coverage/exec.
path
• TakeAllSelector: take all
amplified test
You can also play with
--iteration
27/21