Abstract 2: "Advanced testing in action on a Java project" In 2019 we're all used to writing automated tests in Java projects. It's now time to move up the chain and learn how to implement more complex type of testing. This talk will demonstrate advanced testing practices used by the XWiki open source project (http://xwiki.org), and using Java, Maven, Docker and Jenkins and more: * Testing for backward compatibility with Revapi and an associated strategy * Testing for coverage with Jacoco and defining a viable strategy for slowing improving the situation * Testing the quality of your tests with Descartes Mutation testing * Automatically enriching your test suite with DSpot * Testing various configurations with Docker containers and Jenkins

1. New types of tests for
Java projects
Vincent Massol, January 2020

2. Agenda
•Context
•Backward compatibility testing
•Coverage testing
•Mutation testing
•Configuration testing
•Crash reproduction

3. Context: XWiki
• Open source wiki
• 15 years
• 10-15 active committers
• Very extensible, scripting in wiki pages
• Platform for developing ad-hoc web applications
• Strong build practices using Maven and lots of “Quality” plugins
• Using Jenkins & custom pipeline library for the CI
https://xwiki.org

4. Context: STAMP
• Automatic Test Amplification
• XWiki SAS participating, 3 years
• Experiments on XWiki project
Mutation testing Environment Testing Production Testing

5. Testing Status (after STAMP)
• 11074 automated tests (in 2.7 hours):
• Unit tests (using JUnit5 & Mockito)
• Functional (UI) tests (using Selenium/Webdriver) and using
Docker
Data from 31 Jan 2020

6. New questions
• How can I prevent breaking my users (when I
expose some APIs)? Backward compatibility
• Are my tests testing enough? Coverage
• How good are my tests? Mutation testing
• Do my software work in various setups?
Configuration testing
• How can I reproduce bugs found in production?
Production testing
= in place w/ strategy = in progress

7. Backward Compatibility

8. Backward Compatibility
• For APIs (and SPIs)
• Using the Revapi Maven plugin
• Supports source and binary compatibility
• Strategy:
• Break the build on backward compatibility
violations
• Add ignores in pom.xml if really needed
and ok
• Add ignore list in release notes to warn
users of your APIs

9. Backward Compatibility
• Strategy continued:
• Use @Deprecated annotation
• Once no more code uses deprecated API, move it to Legacy
module.We don’t break backward compatibility!
• Use AspectJ in Legacy module to generate and aspectified
API (JAR)
• Makes code clean and not able to use the Legacy modules
(build-enforced)
• Distribute the legacy modules
public privileged aspect ApiCompatibilityAspect
{
@Deprecated
public boolean Api.checkProgrammingRights()
{
return this.hasProgrammingRights();
}
}
Java 8 default methods helped a lot

10. Backward Compatibility
• Strategy continued:
• For young APIs, use @Unstable + @Since
• Enforced by build (to make sure @Since is there).
Custom checkstyle rule (or Spoon rule)
• Max duration is one cycle (i.e. 1 year).
• Enforced by build (fails the build if beyond).
/**
* ...
* @since 9.7RC1
*/
@Unstable
public List<MacroDescriptor> getMacroDescriptors(Syntax syntax) throws MacroLookupException
{
...
}
Mention that as a result XWiki
extensions and scripts can still run in
XWiki even several years after they
were released.

11. Coverage

12. Test Coverage - Local
• Using Jacoco and Clover
• Strategy - “Ratchet effect”:
• Each Maven module has a threshold
• Jacoco Maven plugin fails if new code has less coverage than
before in %
• Devs are allowed to increase threshold
Of course TPC is not panacea. You
could have 100% and app not
working. Also need functional tests.
Aim for 80%.

13. Test Coverage - Global
• Issue: Local coverage can increase and
global decrease
• Removed code with high TPC
• Code tested indirectly by functional
tests and code refactoring led to
different paths used
• New module with lower TPC than
average
Global TPC evolution

14. Test Coverage - Global
• Strategy:
• Global Clover TPC computed automatically every night on
Jenkins for all repos combined, using a pipeline
• Email sent to developers with report in email (see next slide)
• But only when global TPC is reduced
• Developers fix module they have been working on
• Release Manager (RM) ensures that report passes before
release & we add one step in our Release Plan check list.
Source: http://massol.myxwiki.org/xwiki/bin/view/Blog/ComparingCloverReports

15. Test Coverage - Global

16. Mutation Testing

17. Mutation Testing
• Using PIT/Gregor, PIT/Descartes
• Concepts of PIT
• Modify code under test (mutants) and run tests
• Good tests kill mutants
• Generates a mutation score similar to the coverage %
• Descartes = extreme mutations that execute fast and have high
values
https://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes

18. Mutation - Issue Report

19. Mutation - Example

20. Mutation - Example
result =
(getId() == macroId.getId() || (getId() != null && getId().equals(macroId.getId())))
&& (getSyntax() == macroId.getSyntax() || (getSyntax() != null && getSyntax().equals(
macroId.getSyntax())));

21. Mutation - Example
@Test
public void testEquality()
{
MacroId id1 = new MacroId("id", Syntax.XWIKI_2_0);
MacroId id2 = new MacroId("id", Syntax.XWIKI_2_0);
MacroId id3 = new MacroId("otherid", Syntax.XWIKI_2_0);
MacroId id4 = new MacroId("id", Syntax.XHTML_1_0);
MacroId id5 = new MacroId("otherid", Syntax.XHTML_1_0);
MacroId id6 = new MacroId("id");
MacroId id7 = new MacroId("id");
Assert.assertEquals(id2, id1);
// Equal objects must have equal hashcode
Assert.assertTrue(id1.hashCode() == id2.hashCode());
Assert.assertFalse(id3 == id1);
Assert.assertFalse(id4 == id1);
Assert.assertFalse(id5 == id3);
Assert.assertFalse(id6 == id1);
Assert.assertEquals(id7, id6);
// Equal objects must have equal hashcode
Assert.assertTrue(id6.hashCode() == id7.hashCode());
}
Not testing
for inequality!
Improved thanks to Descartes!

22. Mutation - Strategy
• Been in place for 1 year now
• Fail the build when the mutation score of a given module is below
a defined threshold in the pom.xml
• The idea is that new tests should, in average, be of quality equal or
better than past tests.
• Other idea: hook on CI to run it only on modified code/tests.
General goal with coverage + mutation: maintain quality

23. Mutation - Limitations
• A bit slow, by definition
• 2h30 on xwiki
• Executed in scheduled jobs (once per day)
• PITmp Maven plugin to execute on multi-module
• All sources mutated, not just sources where tests are
• Much slower as a consequence, and thus difficult to
generate a global mutation score

24. Mutation: Going further
• Using DSpot
• Uses PIT/Descartes but injects
results to generate new tests
• Adds assertions to existing tests
• Generate new test methods
• Selector can be PIT/Gregor, PIT/
Descartes, Jacoco (instruction
coverage), Clover (Branch
coverage)
https://massol.myxwiki.org/xwiki/bin/view/Blog/TestGenerationDspot

25. Mutation: Dspot Example 1
public void escapeAttributeValue2() {
String escapedText = XMLUtils.escapeAttributeValue("a < a' && a' < a" => a < a" {");
// AssertGenerator add assertion
Assert.assertEquals("a &#60; a&#39; &#38;&#38; a&#39; &#60; a&#34; =&#62; a &#60; a&#34; &#123;", escapedText);
// AssertGenerator create local variable with return value of invocation
boolean o_escapeAttributeValue__3 = escapedText.contains("<");
// AssertGenerator add assertion
Assert.assertFalse(o_escapeAttributeValue__3);
// AssertGenerator create local variable with return value of invocation
boolean o_escapeAttributeValue__4 = escapedText.contains(">");
// AssertGenerator add assertion
Assert.assertFalse(o_escapeAttributeValue__4);
// AssertGenerator create local variable with return value of invocation
boolean o_escapeAttributeValue__5 = escapedText.contains("'");
// AssertGenerator add assertion
Assert.assertFalse(o_escapeAttributeValue__5);
// AssertGenerator create local variable with return value of invocation
boolean o_escapeAttributeValue__6 = escapedText.contains(""");
// AssertGenerator create local variable with return value of invocation
boolean o_escapeAttributeValue__7 = escapedText.contains("&&");
// AssertGenerator add assertion
Assert.assertFalse(o_escapeAttributeValue__7);
// AssertGenerator create local variable with return value of invocation
boolean o_escapeAttributeValue__8 = escapedText.contains("{");
// AssertGenerator add assertion
Assert.assertFalse(o_escapeAttributeValue__8);
}
Generated test
New test
@Test
public void escapeAttributeValue()
{
String escapedText = XMLUtils.escapeAttributeValue("a < a' && a' < a" => a < a" {");
assertFalse("Failed to escape <", escapedText.contains("<"));
assertFalse("Failed to escape >", escapedText.contains(">"));
assertFalse("Failed to escape '", escapedText.contains("'"));
assertFalse("Failed to escape "", escapedText.contains("""));
assertFalse("Failed to escape &", escapedText.contains("&&"));
assertFalse("Failed to escape {", escapedText.contains("{"));
}
Original test

26. Mutation: Dspot Example 2
Generated test
Original test
Also increase coverage
Before: 70.5%
After: 71.2%

27. Mutation: Dspot Strategy
• DSpot is very slow to execute (between 3 to 20mn on
small modules)
• One strategy is to run it on CI and in the pipeline commit
generated tests in a different source root.
• And run it only on Tests affected by commit changeset
• Configure Maven to add a new test directory source using
the Maven Build Helper plugin.
• Work in progress: small coverage and mutation score
improvements on XWiki so far.

28. Configuration Testing

29. Environment Testing
• Environment = combination of Servlet
container & version, DB & version, OS,
Browser & version
• And also cluster mode, LibreOffice
integration, external SOLR, etc
• Need: Be able to run/debug functional
tests on local dev machines as well as on
CI
• Using Docker / TestContainers

30. Environment Testing
https://massol.myxwiki.org/xwiki/bin/view/Blog/EnvironmentTestingExperimentations

31. Environment Testing

32. Environment Testing
• Feedback: takes about 3 minutes to deploy all (and 1 minute for the
test)
• Strategy
• Run on CI (Jenkins)
• As part of the main pipeline but using the parametrised
scheduler plugin
• 3 jobs
• “latest”: latest version, executes daily
• “all”: all supported configs, executed weekly
• “unsupported”: all currently unsupported configs, executed
monthly

33. Crash Reproduction

34. Crash Reproduction
• Tool: EvoCrash / Botsing
• Concept:Take a stack trace
and generates a test that,
when executed, leads to this
stack trace
• i.e. find the conditions that
leads to the problem

35. EvoCrash Example
10 frames reproduced!

36. Evocrash - Feedback
• Can take a long time to reproduce, doesn’t always succeed
• Generates a test that reproduces the problem, not the fix!
• Often you’d write a test at a different level (usually up in the call
chain, to be more meaningful to the use case)
• Is useful for newcomers who don’t know the codebase well as it
helps pinpoint the problem. Acts as a timesaver.

37. Parting words
• Experiment, push the limit!
• Some other types of tests not covered and that also need
automation
• Performance/Stress testing
• Usability testing
• others?
• Functional UI tests and Config tests are hard because of flickers and
the required environment stability

38. Q&A
Me

39. Vincent Massol
vincent@massol.net
skype: vmassol
http://about.me/vmassol
http://xwiki.org
http://xwiki.com