“Who watches the watchmen": Testando nossos testes - Maria Clara Santana, Picter [JS EXPERIENCE 2018] - 5 de julho de 2018 São Paulo/SP

1. “WHO WATCHES THE
WATCHMEN”
Testing our tests
Maria C. Santana
JS Experience ‘18

2. Maria Clara
• Front-end Developer @ Picter
• Live Coding Instructor @ Udacity
• Likes dogs and sparkling water

3. Types of software
errors
• Construction errors;
• Specification errors;
• Design errors;
• Requirements errors;
Toward a theory of test data selection, Softech Inc.

4. BLACK-BOX
TESTING

5. Input Output

8. •Pro: Allows to check the functionality of a program
without worrying with the structure behind it;
•Con: Does not offer insights about the tested program’s
structure;

9. High code coverage
!==
Effective Test Suite

10. Source: https://gph.is/1jE2Wzg

13. Take it with a grain of salt.

14. “Quis
custodiet ipsos
custodes?”
Satires, 2a.c.

15. WHITE-BOX
TESTING

16. Input Output

19. •Pro: More thorough tests;
•Con: Can be complex and require a lot of work due to
implementation changes;

20. MUTATION
TESTING

21. CONCEPT

22. #1 STEP
Test data Original code

23. •If it fails, then the original program is wrong;
•If it passes, then we should proceed with creating
mutants;

24. #2 STEP
Test data N modified versions
of the original code

25. •#1: given the test data, the modified versions have
different outputs from the original one;
•#2: some (or all of them) versions have the same output
of the original one;

26. MUTATORS TYPES

27. Not this kind, though…

28. AAR (array for array replacement)

29. ABS (absolute value insertion)

30. AOR (arithmetic operator replacement)

31. UOI (unary operation insertion)

32. Many others…
• Array constant replacement;
• Logical connector replacement;
• Relational operator replacement;
• Constant replacement;
• Return statement replacement;
• Statement deletion;
• …

33. EXAMPLE

34. Original program

35. Test suite

36. Mutant #1 - Conditional operator replacement

37. Mutant #1 - Test suite results
Mutant was killed! 🎉

38. Mutant #2 - Conditional operator replacement

39. Mutant #2 - Test suite results
Mutant has survived… ☹

40. EQUIVALENT
MUTANTS

42. Mutant #3 - Return statement replacement

43. Mutant #3 - Test suite results
Mutant was killed! 🎉

44. mutation score = mutants killed / total of
mutants; -> 100%

45. TOOLS

46. https://stryker-mutator.io/

47. Trade-offs
•Open-source;
•Active maintenance and improvements;
•Supports most of the mainstream runners (karma,
jasmine, jest, …);
•Does not support React;

48. Other languages…
• Python: https://github.com/mutpy/mutpy;
• Ruby: https://github.com/mbj/mutant;
• Java: https://github.com/hcoles/pitest;

49. PROS

50. •Application safety;
•Evaluation of edge cases;
•Better test design;

51. CONS

52. •High computational costs;
•Filtering equivalent mutants is still a non-optimal task;

53. Q&A

54. THANKS!
olarclara.github.io