The benefits of BDD (Behaviour-Driven Development)-style automated acceptance tests are huge. Far beyond simply testing your application, BDD uses automated acceptance tests to improve team collaboration and communication, focus development efforts on truly valuable features, and provide meaningful progress reports and reliable feature documentation. However one of the biggest challenges to implementing Automated Acceptance Testing is writing them in a way that will be easy to maintain as the project progresses. Indeed, the cost of maintaining the acceptance test suite should not be more than the value that it provides. This talk explores strategies for writing maintainable and meaningful automated acceptance tests, including aspects such as: Challenges to maintaining automated acceptance tests How to organise and structure your tests more effectively Writing truly meaningful acceptance tests When to test the UI, and when to test the backend How to deal with database setup and teardown How to avoid test fragility How to get the most out of ATDD reporting

1. and the art of
Automated Acceptance Test Suite Maintenance
An Inquiry into Test Automation Quality
ZEN

2. John Ferguson Smart
Consultant
Trainer
Mentor
Author
Speaker
Coder

3. Automated Acceptance Testing is valuable

4. Faster release cycles

5. Improved visibility

6. Reduce risk and uncertainty

7. Automated Acceptance Testing is not easy

8. Fragile?

9. Expensive to maintain?

10. Slow to run?

11. Is it worth it?

12. “It is fun to have fun
But you have to know how”
Sure it is!

13. Three rules of writing
maintainable acceptance tests
A test should only need changing if the subject of
the test changes
Involve the testers early
Report on what requirements you verified,
not what tests you ran

14. Rule #1 - Stability
A test should only change
if the concept under test changes

15. What do ogres and good automated tests have in common?

16. (This test is not like an ogre)

17. The Record-Replay Scam
in theory
in practice

18. Remember...good automated acceptance tests have layers

19. Start off with the business objectives
Business-­‐level
requirements

20. Describe the business flow
High-­‐level
steps
(business
flow)

21. Describe the page interactions
Page-­‐level
interac<ons

22. Page implementation details
HTML
details
go
here

23. Business
Rules
Business
Flow
Page/Component
interac<ons
Page/Component
details
Good automated acceptance tests have layers

24. Business
Rules
Business
Flow
Page/Component
interac<ons
Page/Component
details
Good automated acceptance tests have layers
Only
expose
what
the
business
needs
to
know
Scenario: Register for online banking
Given that Joe wants to register for online banking
When he goes to the registration page
And he enters 'Joe' in the first name field
And he enters 'Bloggs' in the surname field
And he enters 'Joe@bloggs.com" in the email field
And he enters '01/01/1980' in the date of birth field
And he enters '1 George street' in the street field
And he enters 'Sydney' in the city field
And he enters '2000' in the post code field
And he clicks on submit
Then his application should be created in a pending state
And he should be sent a PDF contract to sign by email
Who cares!

25. Business
Rules
Business
Flow
Page/Component
interac<ons
Page/Component
details
Good automated acceptance tests have layers
Only
expose
what
the
business
needs
to
know
Scenario: Register for online banking
Given that Joe wants to register for online banking
When he submits his application online
Then his application should be created in a pending state
And he should be sent a PDF contract to sign by email

26. Business
Rules
Business
Flow
Page/Component
interac<ons
Page/Component
details
Good automated acceptance tests have layers
Centralize
and
reuse
test
logic
Centralize
and
reuse
test
implementa<on
details

27. Business
Rules
Business
Flow
Page/Component
interac<ons
Page/Component
details
Good automated acceptance tests have layers
Use
UI
tests
for
tes<ng
user
interac<ons
with
the
screens
Non-­‐UI
tests
are
OK
for
tes<ng
business
logic
and
calcula<ons

28. How much should I test?
When
in
doubt,
go
broad
and
shallow
Deep
and
narrow
leads
to
waste

29. Rule #2 - Relevance
Report what requirements you verified,
not what tests you ran

30. Feature Coverage
How many requirements are
you really testing?

31. Requirements
Capabilities/Epics
Book
flights
online
Manage
Frequent
Flyer
account
Features
Search
available
flights
Book
flight
with
points
Pay
with
credit
card
View
account
status
Real
<me
no<fica<ons
Status
upgrades
Acceptance Criteria
-­‐
no<fica<ons
for
delayed
flights
-­‐
no<fica<on
for
cancelled
flights
-­‐
no<fica<on
of
baggage
carousel-­‐
Should
see
current
points
-­‐
Should
see
special
offers
-­‐
should
see
current
status
Automated Tests
Given
Joe
is
booked
on
flight
QF108
And
Joe
is
registered
for
SMS
no<fica<ons
When
the
flight
is
delayed
then
Joe
should
be
sent
an
SMS
no<fica<on
Given
Joe
is
booked
on
flight
QF108
And
Joe
is
registered
for
email
no<fica<ons
When
the
flight
is
cancelled
then
Joe
should
be
sent
an
email
no<fica<on
with
an
alterna<ve
flight
Given
Joe
is
booked
on
flight
QF108
And
Joe
is
registered
for
email
no<fica<ons
When
the
flight
is
delayed
then
Joe
should
be
sent
an
email
no<fica<on
? ?
? ?
Given
Joe
is
a
<status>
Frequent
Flyer
with
<current-­‐points>
points
And
he
has
earned
<points>
points
this
month
Then
his
new
status
should
be
<new-­‐status>
status
|
current-­‐points
|
points
|
new-­‐status
SLIVER
|
9800
|
100
|
SILVER
SLIVER
|
9900
|
100
|
GOLD
GOLD
|
49900
|
100
|
PLATINUM

32. How
many
failed
Test
results
can
tell
us...
How
many
tests
passed
How
many
weren’t
run

33. Test
results
can
tell
us...
What
tests
exist
for
a
given
feature
How
stable
the
feature
is

34. Test
results
can
tell
us...
How
a
feature
was
tested

35. Test
results
can
tell
us...
What
a
feature
looks
like

36. Test
results
can’t
tell
us
what
features
were
not
tested
?

37. Requirements
Capabilities/Epics
Book
flights
online
Manage
Frequent
Flyer
Features
Search
available
Book
flight
with
points
Pay
with
credit
card
View
account
Real
<me
no<fica<on
Status
upgrades
Acceptance Criteria
-­‐
no<fica<ons
for
delayed
flights
-­‐
no<fica<on
for
cancelled
flights
-­‐
no<fica<on
of
baggage
carousel-­‐
Should
see
current
points
-­‐
Should
see
special
offers
-­‐
should
see
current
status
Acceptance
criteria
defined
and
tests
working
Acceptance
criteria
defined,
but
no
tests
Acceptance
criteria
defined,
tests
specified
but
not
implemented
Failing
tests
No
acceptance
criteria
defined
Test Results
Feature Coverage

38. Requirements
Capabilities/Epics
Book
flights
online
Manage
Frequent
Flyer
Features
Search
available
Book
flight
with
points
Pay
with
credit
card
View
account
Real
<me
no<fica<on
Status
upgrades
Acceptance Criteria
-­‐
no<fica<ons
for
delayed
flights
-­‐
no<fica<on
for
cancelled
flights
-­‐
no<fica<on
of
baggage
carousel-­‐
Should
see
current
points
-­‐
Should
see
special
offers
-­‐
should
see
current
status
Done
Not
tested
at
all
Tests
not
finished
yet
Broken
Nothing
done
yet
Test Results
Feature Coverage

39. Feature Coverage
High
level
capabili<es

40. Feature Coverage
What
features
are
defined
for
this
capability?

41. Feature Coverage
What
stories
are
defined
for
this
feature?
How
many
stories
have
automated
acceptance
criteria?

42. Feature Coverage
What
stories
are
defined
for
this
feature?
How
many
stories
have
automated
acceptance
criteria?
What
acceptance
criteria
have
been
automated?

43. Rule #3 - Involve the testers early
The most meaningful tests are written
before the application is built

45. Story
bug
reports
Working
code boring
manual
tes<ng
WASTE
How do you build
a feature?

46. Working
code
and
Working
Automated
Acceptance
Tests
Exploratory
tes<ng,
usability
tes<ng...
Shared
understanding
Story
Examples
Automated
acceptance
criteria
How do you build
a feature?

47. Good
automated
acceptance
tests
have
layers
Involve
the
testers
early
Feature
Coverage
tells
you
where
you
are
at
Takeaways

48. To learn more...
h[p://thucydides.info

49. Thank you
John
Ferguson
Smart