This is a tutorial that was presented at: The 20th International Conference on Software and Systems Reuse (ICSR'22) Date of Conference: 15-17 June 2022 Conference Location: Virtual Conference website: https://icsr2022v2.wp.imt.fr/

1. Test Anti-Patterns:
From Definition to Detection
Anthony Peruma
Ph.D. Candidate - Rochester Institute of Technology, USA
Incoming Assistant Professor - University of Hawaii at Mānoa , USA
International Conference on Software and Systems Reuse (ICSR), 15-17 June 2022

2. About Anthony…
Experience/Qualifications
Assistant Professor - University of Hawai‘i
at Mānoa, USA (Starting August 2022)
Ph.D. Candidate - Rochester Institute of
Technology, USA (Expected June 2022)
Masters in Software Engineering -
Rochester Institute of Technology, USA
10+ years of industry experience
Research Interests
Program Comprehension - Identifier Naming
Software Quality - Test Smells
Software Refactoring
Software Maintenance & Evolution
Empirical Software Engineering
https://www.peruma.me
https://twitter.com/ShehanPeruma

3. Agenda
▸ Introduction
▸ Test Smell Types
▸ Test Smell Detection Tools
▸ Conclusion

4. 1.
Introduction

5. Software maintenance is not cheap!
5
I n t r o d u c t i o n
▸A high-quality system need not be necessarily
maintenance-friendly
▸Systems built using poor design/coding practices
can meet functional requirements
▸In the long run, such events impact software
maintenance - and maintenance is not cheap!
▹ Maintenance consumes 50% to 80% of resources

6. Improving the internal quality of the system
6
I n t r o d u c t i o n
▸Refactoring – Process of restructuring the code to improve its
internal structure while preserving its external behavior
▹ Correcting an inherently flawed design
▹ Improves non-functional attributes – readability, reusability, maintainability, etc.
▹ In theory, refactoring is not about fixing bugs or making functional updates –
refactoring is updating to code to make it easier to fix bugs and update functionality
▸How do you know when its time to refactor? – Look for the
occurrence of “smells” in the code
▹ Violations of software development fundamentals that reduce the quality of code
▸Smells make code harder to understand and make it more
prone to bugs and changes – indicator of a deeper problem

7. Towards maintenance-friendly code
7
I n t r o d u c t i o n
▸Researchers and industry have defined and created
approaches and tools to detect code in need of
refactoring
▹ Design/code smells - Cohesion, Coupling, God Class, etc.
▹ Tools - FindBugs, PMD, Checkstyle, etc.
▸Catalog of standard/well-established operations to correct
(i.e., refactor) specific design flaws (i.e., smells)
▸Research and tools have been primarily on production code

8. Examples of code smells
8
I n t r o d u c t i o n
▸Long Method – makes readability difficult
▹ Possible Fix: Extract Method
▸Long Class – indicator of duplicate code or unnecessary
functionality
▹ Possible Fix: Extract Class or Extract Subclass
▸Long Parameter List – hard to read and requires frequent
changes
▹ Possible Fix: Replace Parameter with Object
More : https://sourcemaking.com/refactoring/smells

9. What are test smells?
9
I n t r o d u c t i o n
▸Test code, like production code, is subject to smells
▹ Majority of the smells in test code are different from traditional code smells
▸Test smells are sub-optimal design choices developers make
when implementing test cases
▸These are usually violations in X-Unit guidelines and concepts
▸Formally introduced in 2001 with 11 smell types
▸Inclusion of additional smell types through the years by the
research and developer community

10. Test smell research
10
I n t r o d u c t i o n
▸Most studies are on traditional Java applications
▹ Other programming languages: Python, C++, Smalltalk, Scala
▸Most are empirical studies on open-source systems
▹ Studies include: new smell types, evolution and longevity, smell detection tools,
refactoring approaches, literature reviews/mapping, and developer surveys
▸Research findings on test smells include:
▹ Smells negatively impact code comprehension of the test suite
▹ Smells increase the change- and defect- proneness of the test suite
▹ Specific smells can result in flaky tests – i.e., fails to produce the same result each time
it is executed
▹ Smells are subjective – not all developers consider specific smells to be harmful and
correcting smells might not be feasible, especially in well-established systems

11. 2.
Test Smell Types

12. Test smell types
12
T e s t S m e l l T y p e s
▸ 60+ test smell types
▸ Applicable to Java, Python, C++ & Smalltalk
▹ Just because there aren’t defined smell types for other programming languages, doesn’t
mean they don’t exist! – scope for future research!
▸ Most smells are related to JUnit
▸ Some smell types are applicable to more than one
programming language or testing framework

13. Some common smell types
13
T e s t S m e l l T y p e s
▸ Assertion Roulette
▸ Conditional Test Logic
▸ Constructor Initialization
▸ Default Test
▸ Duplicate Assert
▸ Eager Test
▸ Empty Test
▸ General Fixture
▸ Ignored Test
▸ Lazy Test
▸ Magic Number Test
▸ Mystery Guest
▸ Redundant Print
▸ Redundant Assertion
▸ Sleepy Test
▸ Unknown Test

14. Assertion Roulette
14
T e s t S m e l l T y p e s
▸ Occurs when a test method has multiple non-documented assertions
▸ Challenging to understand the reason for the failure of the test, in an
automated integration build, without an assertion message
S o l u t i o n :
• Break up the test method (i.e., Extract
Method) into independent test cases
• Add assertion explanation

15. Conditional Test Logic
15
T e s t S m e l l T y p e s
▸ Conditions within the test
method will alter the
behavior of the test and its
expected output
▸ Lead to situations where the
test fails to detect defects in
the production method
▸ Can negative impact code
comprehension activities S o l u t i o n :
• Break up the test method logic (i.e.,
Extract Method) into independent test
cases

16. Constructor Initialization
16
T e s t S m e l l T y p e s
▸ Ideally, the test suite should
not have a constructor
▸ Initialization of fields should
be in the setUp() method
(i.e., test fixtures)
▸ Developers unaware of test
fixtures enable this smell
S o l u t i o n :
• Replace constructor with setup()
Note: Beware of introducing the General
Fixture smell

17. Default Test
17
T e s t S m e l l T y p e s
▸ Android Studio specific
smell -- default test class
meant to serve as an
example
▸ Should be removed as it
serves no concrete purpose
and may lead to confusion
▸ A test-first approach will
force developers to remove
the file S o l u t i o n :
• Remove from project or rename and
remove example test

18. Duplicate Assert
18
T e s t S m e l l T y p e s
▸ The same condition is
tested multiple times within
the same test method
▸ The name of the test
method should be an
indication of the test
▸ Possible situations –
grouping multiple conditions
into a single test; debugging;
copy-paste S o l u t i o n :
• Break up the test method (i.e., Extract
Method) into independent test cases

19. Eager Test
19
T e s t S m e l l T y p e s
▸ When a single test verifies too much functionality
▸ Invocation of several production methods from a single test method
▸ Most likely due to developers reducing the number of tests
S o l u t i o n :
• Break up the test method (i.e., Extract
Method) into independent test cases
(one for each production method)

20. Empty Test
20
T e s t S m e l l T y p e s
▸ When a test method has no
executable statements
▸ JUnit will indicate that the
test passes even if there are
no executable statements
present in the method body
▸ Behavior-breaking changes
will not be caught if the test
case is always passing
S o l u t i o n :
• Remove the test method
Note: Ignoring the test case will result in the
Ignore Test smells

21. General Fixture
21
T e s t S m e l l T y p e s
▸ Occurs when a test case fixture
is too general and the test
methods only access part of it.
▸ A test setup method that
initializes fields that are not
accessed by test methods
indicates that the fixture is too
generalized
▸ Being too general results in
unnecessary work is being done
when a test method is run
S o l u t i o n :
• Perform an Extract Method operation
on setup() to create a new method
that contains variables not used by
test cases

22. Ignored Test
22
T e s t S m e l l T y p e s
▸ JUnit4 provides developers
with the ability to suppress
test methods from running
▸ Ignored test methods result
in overhead with regards to
compilation time and an
increase in code complexity
and comprehension
S o l u t i o n :
• Remove the test method from the
class

23. Lazy Test
23
T e s t S m e l l T y p e s
▸ Occurs when
multiple test
methods invoke
the same
production method
▸ Changes to the
production code
can lead to
multiple changes
in the test file S o l u t i o n :
• Check and remove duplicate tests
• Inline variable to setUp()

24. Magic Number Test
24
T e s t S m e l l T y p e s
▸ Test method contains
unexplained and
undocumented numeric
literals
▸ Use of magic numbers over
constants negatively
impacts code readability and
understandability
S o l u t i o n :
• Replace number with symbolic
constant

25. Mystery Guest
25
T e s t S m e l l T y p e s
▸ Occurs when a test method
utilizes external resources
(e.g., files, database, etc.).
▸ Not self-contained (i.e., is
dependent on external
resource for success)
▸ Might result in stability and
performance issues
S o l u t i o n :
• Use mock objects in place of external
resources

26. Redundant Assertion
26
T e s t S m e l l T y p e s
▸ Assertion statements that
are either always true or
false
▸ Common reason for the
existence of this smell is
due to developer mistakes
▸ Might exist to support edge
cases
S o l u t i o n :
• Change parameters in the assertion
method

27. Redundant Print
27
T e s t S m e l l T y p e s
▸ Unit tests are executed as
part of an automated script
▸ They can consume
computing resources or
increase execution time
▸ A common reason for the
existence of this smell is
due to developer debugging
S o l u t i o n :
• Remove print statement

28. Sleepy Test
28
T e s t S m e l l T y p e s
▸ Developers introduce this
smell when they need to
pause execution of
statements in a test method
for a certain duration
▸ Explicitly causing a thread to
sleep can lead to
unexpected results as the
processing time for a task
differs when executed in
various environments and
configurations
S o l u t i o n :
• No definitive solution for this smell
type

29. Unknown Test
29
T e s t S m e l l T y p e s
▸ The assertion statement
helps to indicate the
purpose of the test
▸ JUnit will show the test
method as passing
▸ Missing assertions will
hinder understandability of
the test
▸ Usually introduced due to
mistakes
S o l u t i o n :
• Include assert method in the test
method

30. 3.
Test Smell Detection Tools

31. Automated detection of test smells
31
T e s t S m e l l T y p e s
▸ 20+ test smell detection tools
▹ There is an overlap in the types of smells the tools detect
▹ However, not all tools are under active development
▹ Earliest tool was released in 2006
▹ 2019 and 2020 had the greatest number of tools developed – 4 in each year
▹ Not all tools publish details about their correctness
▹ Only a handful of tools have been utilized in multiple empirical studies by the community
▹ Some tools can be integrated into a code monitoring system (e.g., automated build system)
▸ Automatic detection of test smells in Java, Scala, Smalltalk, and
C++ test suites
▹ JUnit (Java) is the most common test framework supported by the tools – 18 tools
▸ Tools are available as IDE plugins or command-line interface
▹ InteilliJ and Eclipise plugins

32. TSDetect
▸ Open-source, Java-based,
static analysis
▸ Available as a standalone
jar and requires a list of file
paths as input
▸ Utilizes an abstract syntax
tree to parse and detect
test smells
▸ Detects 19 test smells (12
proposed + 7 existing)
▸ Average F-Score of 96.5%
32
T e s t S m e l l D e t e c t i o n T o o l s
High-level architecture of TSDetect
Peruma et al. tsDetect: An open source test smells detection tool. In
Proceedings of the 28th ACM Joint Meeting on European Software Engineering
Conference and Symposium on the Foundations of Software Engineering

33. PyNose
▸ Open-source, test smell
detector for Python
▸ Available as a plugin for
PyCharm IDE
▹ Results shown within the IDE
▸ Static analysis detection
technique
▸ Detects 15 test smell types
▹ Provides Quick Fix for some types
▹ Apply the fix from within the IDE
▸ Average F-Score of 94.9%
33
T e s t S m e l l D e t e c t i o n T o o l s
Evaluation of PyNose
Wang et al. PyNose: A Test Smell Detector For Python. In 2021 36th IEEE/ACM
International Conference on Automated Software Engineering (ASE). IEEE.

34. Demo
▸ TSDetect
▸ PyNose
34
T e s t S m e l l D e t e c t i o n T o o l s

35. Screenshots - PyNose
35
https://github.com/JetBrains-Research/PyNose

36. 4.
Conclusion

37. Summary
37
C o n c l u s i o n
▸ Test smells are deviations from
X-Unit violations and guidelines
▸ Smelly tests can negatively
impact maintenance lead to
quality issues
▸ There are multiple types of test
smells – most target JUnit
▸ Availability of test smell
detection tools

38. Additional sources
38
C o n c l u s i o n
▸ https://testsmells.org/
▸ XUnit Test Patterns Refactoring Test Code:
▹ Meszaros, G. (2007). xUnit test patterns: Refactoring test code. Pearson Education.
http://xunitpatterns.com/index.html
▸ Test smell detection tools:
▹ Aljedaani et al. Test Smell Detection Tools: A Systematic Mapping Study. The International
Conference on Evaluation and Assessment in Software Engineering, June 21–23, 2021.

39. Thanks!
Anthony Peruma
https://www.peruma.me