The document summarizes Atlassian's journey to improve their automated test suite over 2.5 years. They had over 20,000 slow and fragile tests that resulted in dispirited developers accepting failures as normal. Through efforts like restructuring tests, parallelizing execution, removing flaky tests, and splitting the codebase into modules, they were able to reduce build times from over 2 hours to under 1 hour and make passes the norm. Continuous measurement, removing technical debt, and prioritizing test performance were key to their success in improving development feedback loops.

1. #atlassian

2. Heavenly Hell
Automated Tests at Scale
WOJCIECH SELIGA • SENIOR DEV MANAGER • ATLASSIAN • @WSELIGA

3. About me
• Coding since 6 yo
• Agile Practices (inc. TDD) since 2003
• Dev Nerd, Tech Leader, Agile Coach,
Speaker, PHB
• 7 years with Atlassian
(JIRA Senior Dev Manager)
• Spartez Co-founder & CEO

4. XP Promise
Cost of Change
Waterfall
XP
Time

5. XP Promise
Cost of Change
Waterfall
XP
Time

6. The Story

7. About 2.5 years ago

8. Almost 10 years
of accumulating
legacy automatic tests

9. About 20 000 tests
on all levels of abstraction
*just in core JIRA

10. Very slow (even hours)
and fragile feedback loop

11. Serious performance and
reliability issues

12. Dispirited devs
accepting RED as a norm

13. Feedback
Speed
`
Test
Quality

14. Design
Restructure
Share
Respect
Prune
Test Code is Not Trash
Refactor Maintain
Review
Discuss
Rewrite

15. Test Pyramid
Selenium
REST / HTML Tests
Unit Tests (including QUnit)
Fastest, lowest overall confidence
Slowest, highest overall confidence

16. Selenium
REST / HTML Tests
Unit Tests (including QUnit)
Test Pyramid
90%
9%
1%

17. Optimum Balance

18. Optimum Balance
Isolation

19. Optimum Balance
Isolation Speed

20. Optimum Balance
Isolation Speed Coverage

21. Optimum Balance
Isolation Speed Coverage Level

22. Optimum Balance
Isolation Speed Coverage Level Access

23. Optimum Balance
Isolation Speed Coverage Level Access Effort

24. Dangerous to temper with

25. Dangerous to temper with
Quality / Determinism

26. Dangerous to temper with
Quality / Determinism Maintainability

27. Almost two years later…

28. People

29. People - Motivation
Making GREEN the norm

30. Shades of Red

31. Build Tiers and Policy
Tier A1 - green soon after all commits
unit tests and functional* tests
Tier A2 - green at the end of the day
WebDriver and bundled plugins tests
Tier A3 - green at the end of the iteration
supported platforms tests, compatibility tests

32. Wallboards:
Constant
Awareness

33. Training
• Favouring assertThat over assertTrue/False and assertEquals
• Avoiding races - Atlassian Selenium with its TimedElement
• Favouring unit tests over functional tests (including QUnit
over WebDriver)
• Promoting Page Objects
• Brownbags, blog posts, code reviews

34. Quality

35. Re-run failed tests and see if they pass
Automatic Flakiness Detection
Quarantine

36. Quarantine - Healing

37. SlowMo - expose races

38. Selenium 1

39. Selenium 1

40. Selenium ditching
Sky did not fall in

41. Ditching - benefits
• Freed build agents - better system throughput
• Boosted morale
• Gazillion of developer hours saved
• Money saved on infrastructure

42. Ditching - due diligence
• conducting the audit - analysis of the coverage we lost
• determining which tests needs to rewritten (e.g. security related)
• rewriting the tests (good job for new hires + a senior mentor)

43. Flaky Browser-based Tests
Races between test code and asynchronous page logic
Playing with "loading" CSS class does not really help

44. Races Removal with Tracing
// in the browser:!
function mySearchClickHandler() {!
doSomeXhr().always(function() {!
// This executes when the XHR has completed (either success or failure)!
JIRA.trace("search.completed");"
});!
}!
// In production code JIRA.trace is a no-op
// in my page object:!
@Inject!
TraceContext traceContext;!
!
public SearchResults doASearch() {!
Tracer snapshot = traceContext.checkpoint();!
getSearchButton().click(); // causes mySearchClickHandler to be invoked!
// This waits until the "search.completed"
// event has been emitted, *after* previous snapshot !
traceContext.waitFor(snapshot, "search.completed"); !
return pageBinder.bind(SearchResults.class);!
}!

45. Speed

46. Can we halve our build times?

47. Parallel Execution - Theory
End of Build
Batches
Start of Build

48. Parallel Execution
End of Build
Batches
Start of Build

49. Parallel Execution - Reality Bites
End of Build
Agent
availability
Batches
Start of Build

50. Dynamic Test Execution
Dispatch - Hallelujah

51. Dynamic Test Execution
Dispatch - Hallelujah

52. "You can't manage what
you can't measure."
not by W. Edwards Deming

53. If you believe just in it
you are doomed.
"You can't manage what
you can't measure."
not by W. Edwards Deming

54. You can't improve the system
if you can't measure it

55. You can't improve the system
if you can't measure it
Profiler, Build statistics, Logs, statsd → Graphite

56. Compilation
Packaging
Executing Tests
Anatomy of Build*

57. Fetching Dependencies
Compilation
Packaging
Executing Tests
Anatomy of Build*

58. Fetching Dependencies
Compilation
Packaging
Executing Tests
Anatomy of Build*
*Any resemblance to maven build is entirely accidental

59. Fetching Dependencies
Compilation
Packaging
Executing Tests
SCM Update
Anatomy of Build*
*Any resemblance to maven build is entirely accidental

60. Agent Availability/Setup
Fetching Dependencies
Compilation
Packaging
Executing Tests
SCM Update
Anatomy of Build*
*Any resemblance to maven build is entirely accidental

61. Agent Availability/Setup
Fetching Dependencies
Compilation
Packaging
Publishing Results
Executing Tests
SCM Update
Anatomy of Build*
*Any resemblance to maven build is entirely accidental

62. Compilation (7min)
JIRA Unit Tests Build

63. Compilation (7min)
Packaging (0min)
JIRA Unit Tests Build

64. Compilation (7min)
Packaging (0min)
Executing Tests (7min)
JIRA Unit Tests Build

65. Compilation (7min)
Publishing Results (1min)
Packaging (0min)
Executing Tests (7min)
JIRA Unit Tests Build

66. Compilation (7min)
Publishing Results (1min)
Packaging (0min)
Executing Tests (7min)
Fetching Dependencies (1.5min)
JIRA Unit Tests Build

67. Compilation (7min)
Publishing Results (1min)
Packaging (0min)
Executing Tests (7min)
SCM Update (2min)
Fetching Dependencies (1.5min)
JIRA Unit Tests Build

68. Agent Availability/Setup (mean 10min)
Compilation (7min)
Publishing Results (1min)
Packaging (0min)
Executing Tests (7min)
SCM Update (2min)
Fetching Dependencies (1.5min)
JIRA Unit Tests Build

69. Decreasing test
execution time to
ZERO
alone would not let
us achieve our goal!

70. Agent Availability/Setup
• starved builds due to
busy agents building
very long builds
• time synchronization
issue - NTPD problem

71. SCM Update - Checkout time
• Proximity of SCM repo
• shallow git clones are not so fast and lightweight + generating extra
git server CPU load
• git clone per agent/plan + git pull + git clone per build (hard links!)
• Much less load on Stash server (no need to queue up)

72. SCM Update - Checkout time
• Proximity of SCM repo
• shallow git clones are not so fast and lightweight + generating extra
git server CPU load
• git clone per agent/plan + git pull + git clone per build (hard links!)
• Much less load on Stash server (no need to queue up)
2 min → 5 seconds

74. Fetching Dependencies
• Fix Predator
• Sandboxing/isolation agent trade-off:
rm -rf $HOME/.m2/repository/com/atlassian/*
into
find $HOME/.m2/repository/com/atlassian/
-name “*SNAPSHOT*” | xargs rm
• Network hardware failure found
(dropping packets)

75. Fetching Dependencies
• Fix Predator
• Sandboxing/isolation agent trade-off:
rm -rf $HOME/.m2/repository/com/atlassian/*
into
find $HOME/.m2/repository/com/atlassian/
-name “*SNAPSHOT*” | xargs rm
• Network hardware failure found
(dropping packets)
1.5 min → 10 seconds

76. Compilation
• Restructuring multi-pom maven project and dependencies
• Maven 3 parallel compilation FTW!
-T 1.5C
*optimal factor thanks to scientific trial and error research

77. Compilation
• Restructuring multi-pom maven project and dependencies
• Maven 3 parallel compilation FTW!
-T 1.5C
*optimal factor thanks to scientific trial and error research
7 min → 1 min

78. Unit Test Execution
• Splitting unit tests into 2 buckets: good and legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
3000 poor tests
(5min)
11000 good tests
(1.5min)
Rewritten entirely
over next year

79. Unit Test Execution
• Splitting unit tests into 2 buckets: good and legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
3000 poor tests
(5min)
11000 good tests
(1.5min)
7 min → 5 min
Rewritten entirely
over next year

80. Functional Tests
• Selenium 1 removal did help
• Faster reset/restore (avoid unnecessary stuff, intercepting SQL
operations for debug purposes - building stacktraces is costly)
• Restoring via Backdoor REST API (JIRA TestKit)
• Using REST API for common setup/teardown operations

81. Functional Tests

82. Publishing Results
• Server log allocation per test → using now Backdoor
REST API (was Selenium)
• Bamboo DB performance degradation for rich build
history

83. Publishing Results
• Server log allocation per test → using now Backdoor
REST API (was Selenium)
• Bamboo DB performance degradation for rich build
history
1 min → 40 s

84. Unexpected Problem
• Stability Issues with our CI server (hardware)
• The bottleneck changed from I/O to CPU
• Too many agents per physical machine

85. Compilation (1min)
JIRA Unit Tests Build Improved

86. Compilation (1min)
Packaging (0min)
JIRA Unit Tests Build Improved

87. Compilation (1min)
Packaging (0min)
Executing Tests (5min)
JIRA Unit Tests Build Improved

88. Compilation (1min)
Packaging (0min)
Publishing Results (40sec)
Executing Tests (5min)
JIRA Unit Tests Build Improved

89. Compilation (1min)
Packaging (0min)
Publishing Results (40sec)
Executing Tests (5min)
Fetching Dependencies (10sec)
JIRA Unit Tests Build Improved

90. Compilation (1min)
Packaging (0min)
Publishing Results (40sec)
Executing Tests (5min)
SCM Update (5sec)
Fetching Dependencies (10sec)
JIRA Unit Tests Build Improved

91. Agent Availability/Setup (3min)*
Compilation (1min)
Packaging (0min)
Publishing Results (40sec)
Executing Tests (5min)
SCM Update (5sec)
Fetching Dependencies (10sec)
JIRA Unit Tests Build Improved

92. Improvements Summary
Tests Before After Improvement %
Unit tests 29 min 17 min 41%
Functional tests 56 min 34 min 39%
WebDriver tests 39 min 21 min 46%
Overall 124 min 72 min 42%
* Additional ca. 5% improvement expected once new git clone
strategy is consistently rolled-out everywhere

93. Better speed increases
responsibility
Fewer commits (authors) per single build
vs.

94. The Quality Follows

95. The Quality Follows

96. The Quality Follows

97. But that's still bad
We want CI feedback loop in a few minutes maximum

98. Splitting The Codebase

99. Inevitable Split - Fears
• Organizational concerns - understanding, managing,
integrating, releasing, coordinating
• Mindset change - if something worked for 10+ years why
to change it?
• Trust - does this library still work?
• We damned ourselves with big buckets for all tests -
where do they belong to?

100. Splitting code base
• Step 0 - JIRA Importers Plugin (3.5 years ago)
• Step 1- New Issue View and Navigator
• Step 2 - now everything else follows (e.g. Workflow Designer)
JIRA 6.0

101. Getting back from hell to heaven is difficult.
Hell sucks in your soul.

102. Key takeaways:
• Visibility and problem awareness help
• Maintaining huge testbed is difficult and costly
• Measure the problem - to baseline
• No prejudice - no sacred cows
• Automated tests are not one-off investment, it's a continuous journey
• Performance is a damn important feature
#atlassian

103. Test performance
is a damn
important feature!

104. XP vs Sad Reality
Cost of Change
Waterfall
XP - ideal
Sad Reality
Time

105. Images - Credits
• Green Traffic Light - by flrnt, CC-BY-SA-2.0
• Turtle - by Jonathan Zander, CC-BY-SA-3.0
• Loading - by MatthewJ13, CC-SA-3.0
• Merlin Tool - by L. Mahin, CC-BY-SA-3.0
• Flashing Red Light - by Chris Phan, CC BY 2.0
• In Heaven - by Daniel Pascoal, CC BY-NC-ND 2.0

106. Thank you!
WOJCIECH SELIGA • SENIOR DEV MANAGER • ATLASSIAN • @WSELIGA