How to get the most out of your team by making developers and testers jointly responsible for running tests.

1. Bridging the Gap
Creating better quality software
using Agile Acceptance Testing
and Fitnesse
Clare McLennan
clare.mclennan@gmail.com
http://crazy-it-adventures.blogspot.com
http://twitter.com/claremclennan

2. Contents
• The Challenge
• What means Testability?
• Fitnesse
• 0 to 90% in 9 months
• Our System Test Toolbox
• Success Factors

3. The Challenge

4. The Challenge
• In the beginning, time to market was
everything
• As our customer base grew it became
important that the system always worked
• Big culture change
• 99.99% uptime target
• Releases weekly
• System was hard to test

5. What means testability?

6. Testability
• Any working system is testable
• Aim for Easy to Test
• How much
– Setup
– Knowledge
is needed to test one aspect of the system?
• See podcast, Test Driven Development is
Design

7. Symptoms of Low Testability
• Frustrating and slow to test anything
• Testers needing continuous help from
developers
• Developers may believe testers are stupid
• Developers avoid system testing and stop after
unit testing succeeds
• Poor understanding of the System develops
• Can’t easily introduce new people to the project

8. Automated
Testing
System
Test
System Predict-
Knowledge ability
Debugging Status
Methods Installer Tools

9. System Test
System Predict-
Knowledge ability
Auto
Debugging Status Tools Test
Methods
Installer

10. How We Improved Testability
• Created an installer so everyone runs the
system in the same way
• Created a means to query the system for
when processing is finished
• Added business time to the system so we
instantly test functionality that takes long
periods of time
Example: Conversion tracking works over a
period of 30 days.

11. Fitnesse

12. Fitnesse in a Nut Shell
• Means of capturing requirements as tests
• Tests turn green if passed, red if failed
• Requirements stay up to date
• Customer or testers write the tests
• Programmers write fixture code to make
the tests run

13. Running a Suite of Tests

14. The Technical Side
• Fitnesse is a wiki
• Recommend to store tests with the code
• Use SLIM (has replaced FIT)
• Java, C++, Ruby, Python and more
• Test fail when testers write them
• Testers can reuse fixtures to create more
tests

15. Sample Fixture

16. Sample Fixture

17. Sample Fixture

18. Sample Fixture Code

20. Script Fixture

21. Query Fixture

22. Test Organisation
• Tests organised into heirachy of suites
• SetUp and TearDown run before each test
• Test History of success/failures
• Tests can have explanatory text
• Fixture toolbox documentation

23. How To Write Good Tests
• Use user language, not programmer
mumbo-jumbo
• Make each test specific
• Write cases not scripts - you should only
specify things relevant for this example
• Generally, if you can’t do it manually you
won’t be able to automate it.
• See http://www.concordion.org/Technique.html

24. Evolution of Our Tests (1)
Test uses magic
numbers from
database – can't
see what this test
is about

25. Evolution of Our Tests (2)
All this setup...

26. Evolution of Our Tests (3)
...replaced by
this
This is a system
current
implementation
detail – not a
requirement

27. Evolution of Our Tests (4)
Hang on, these just
set up data and don't
test anything!
We still aren't finished..
Ah better..

28. 0 to 90% in 12 months

29. Creating Quality Processes
Preprocesses
• First Fitnesse tests were written to prove it
was possible
• First testers joined the project
But...
• Writing automated tests didn't catch on

30. Creating Quality Processes
Stage 1
• QA group was formed to
– Recruite and train testers
– Write and program the Fitnesse automated
tests
– Test new functionality
But...
• QA group struggled to keep up with
development effort

31. Creating Quality Processes
Stage 2
• QA group continued to write tests.
• QA group responsible for running tests
• New Fixture requests were handed over to
development team at start of sprint
But...
• QA group still needed needed system
programmers knowledge to keep tests
working
• Hard to specify upfront all fixtures required
• Programmers hated writing fixtures

32. Creating Quality Processes
Stage 3
• Testers joined in development teams.
• Testers responsibility to write tests
• Dev teams responsibility to get tests
running
• Dev team given a test box to run tests on
• Weekly QA meeting for testers to share
changes and ideas

33. Creating Quality Processes
Stage 3
Highly successful!!
After initial teething in sprint (3 weeks)
everyone was positive about the change

34. What Happened During Stage 3?
• DBA sped up tests
• We reduced the number of GUI
functionality tests required because of
good unit test coverage
• Many manual testing issues were resolved
• Finally testing and development occurred
at the same pace
• Programmers embraced writing tests as
part of their job to maintain quality

35. Change in Testers Role
• More about ensuring good specifications
to prevent bugs
• More testing time spent on exploratory
testing
• Better relationships with programmers
• Less dull work
• More influence on how the system is
written to make testing easier

36. Our System Test Toolbox

37. Our System Test Toolbox
• Ask User fixture
• Business time
• Staged Deployment
• Separate Functionality, Gui Functionality,
Gui Layout, Load, Full System, Sanity and
Full system tests
• Close communication between testers and
programmers to find optimal test strategies

38. Ask User

39. Ask User
• Mix and match human and automated
processes
• Allows tests to be written and run before
all automation is ironed out
– Example: Gui testing will eventually be
automated with Selenium
• User created objects can be referred to in
Fitnesse tests
• Simple idea but really practical!

40. Business Time
• Changes the current time in the system
• Allows testing of scenarios that take a long
time
• Only in testing mode
• Low risk, as if forgotten system still works
correctly in production environment

41. Staged Deployment
• Think Beta testing
• Test throughly, then do a partial release to
– Only some customers, or
– For a small proportion of the daily
impressions, or
– Run old and new system side by side
• Gives a more accurate test

42. Types of testing
Unit – Programmers tests. Pinpoint bugs
quickly.
Functionality – Pure, automatic testing of
the whole system. All processes triggered
Gui Functionality – Test functionality of Gui
page at a time. All other objects required
are generated as for functionality testing

43. Types of Testing
Gui Layout – Pure gui test of layout,
updating.
Load – Use real database, or extra large
database
Full System – Main functionality,with
processes on timers, as in production
Sanity Check – Final check performed
before a each release

44. Success Factors

45. Outcomes
• Automated testing of the system takes one
hour, plus some quick manual tests
• Programmers attitude has changed from
expecting outsiders to validate the system,
to sharing responsibility for this task
• System more easy to test, deploy, monitor
and manage
• A self-running QA process which is
continually improved by development
teams

46. Success Factors
• Realistic time frame (12 months)
• Treated building automated testing system
as a project of it’s own
• Influence over design of system to be
tested
• Started with high ROI, hard to manual test,
functionality
• Get tests working, then perfect
• Gently challenged company culture

47. Future Improvements?
• Automate GUI functionality tests with
Selenium
• Start sprints with a Specification Workshop
• Improve Load tests - run them on a cloud
• Move system knowledge from Fitnesse,
back into the system
• Make tests more user orientated

48. To learn more read
Bridging the
Communications
Gap
by Gojko Adzic

49. References
• Fitnesse http://www.fitnesse.org
• Bridging the Communications Gap by Gojko
Adzic
• Test Driven Development is Design - The Last
Word on TDD, Hansel Minutes Podcast
starring Scott Bellware and Scott Hanselman
• Hints and Tips [for writing acceptance tests] by
David Peterson
http://www.concordion.org/Technique.html