Functional testing patterns
•
2 likes•4,895 views
Software teams mostly find themselves working with three broad categories of tests - unit, integration and functional (excluding technology verification test categories like performance, load, stress etc.). Unit tests indicate whether the code is doing things right. Functional tests are complementary to - but quite different from unit tests. Functional tests tell whether the completed application is working correctly and providing the proper functionality. Simply put, unit tests are written from the code developer's perspective, while functional tests are written from the end user's perspective. When they work reliably, functional tests give users, stakeholders and developers confidence that the software meets agreed upon requirements. In reality, a lot of teams find themselves grappling with perennially failing, hard to understand, slow running tests, which take herculean efforts to maintain while inspiring low confidence in the reliability of the end product. This article examines recipes on how to create and maintain a smoothly running suite of functional/acceptance tests that can be reliably used to verify that the software is ready for release.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Functional testing patterns
Similar to Functional testing patterns (20)
Recently uploaded
Recently uploaded (20)
Functional testing patterns
- 1. Functional Testing Pa0erns Premanand Chandrasekaran ThoughtWorks, Inc. premanand@thoughtworks.com
- 2. Types of Tests Business Facing Support Programming Functional Tests Showcases Acceptance Tests Exploratory Tests Critique Product Usability Tests Unit Tests Performance Tests Component Tests Stress Tests System Tests Security Tests Technology Facing h0p://www.exampler.com/old-‐‑blog/2003/08/21/#agile-‐‑testing-‐‑project-‐‑1 © 2012 ThoughtWorks, Inc. 2
- 3. Functional Tests -‐‑ The Pitch • What Are Functional Tests? o Verification that business requirements are met o Black Box o Automated • Why Functional Tests? o Maintain (high) external quality o Allow team to be bolder o Allow team to go faster © 2012 ThoughtWorks, Inc. 3
- 4. Functional Tests – The Reality • Flaky • Take long time to run • Cumbersome to maintain • Don’t catch many bugs • Some things are hard to automate • Don’t quite give you the feedback you were hoping for © 2012 ThoughtWorks, Inc. 4
- 5. Consequences • Low confidence in automation • Over-reliance on manual testing • Slipped Deadlines AND/OR • Lesser Testing AND/OR • Testing in Production :-) © 2012 ThoughtWorks, Inc. 5
- 6. How To Get Be0er? Technology Process People © 2012 ThoughtWorks, Inc. 6
- 7. Process © 2012 ThoughtWorks, Inc. 7
- 8. Functional Test – Example © 2012 ThoughtWorks, Inc. 8
- 9. Example -‐‑ Specification © 2012 ThoughtWorks, Inc. 9
- 10. Example – Implementation © 2012 ThoughtWorks, Inc. 10
- 11. Implementation Problems? • No correlation between specification and implementation • Intent lost in translation • Easy to write, hard to read, maintain • Too monolithic, lacks abstraction • Causes non-technical domain experts to tune out of the process © 2012 ThoughtWorks, Inc. 11
- 12. Executable Specifications Specification Step Tight Correlation Implementation Step © 2012 ThoughtWorks, Inc. 12
- 13. Functional Test -‐‑ Specification © 2012 ThoughtWorks, Inc. 13
- 14. More useful information in failure reports © 2012 ThoughtWorks, Inc. 14
- 15. Acceptance Criteria DSL • Given some initial context • When an event occurs • Then ensure outcome • Example – © 2012 ThoughtWorks, Inc. 15
- 16. Functional Test – New Implementation © 2012 ThoughtWorks, Inc. 16
- 17. Good Acceptance Criteria Scenario: Advanced Search for a hat! ! Given I am searching on "http://www.etsy.com"! When I click on the select box with css class "select.handmade"! And I select "Knitting"! And I click on the text box with id "#search_query"! And type in "hat"! Then there are search results! Focus on WHAT, not HOW Scenario: Advanced Search for a hat! ! Given I am searching on Etsy.com! And I am not logged in! And I specify the Knitting sub category! And I search for hat! Be@er Then there are search results! ! © 2012 ThoughtWorks, Inc. 17
- 18. How Many Tests? Manual Testing 10% Functional Tests Integration Tests 20% Unit Tests 70% h0p://blog.mountaingoatsoftware.com/tag/agile-‐‑testing © 2012 ThoughtWorks, Inc. 18
- 19. What About Legacy Systems? • Start with functional tests • Cover with unit tests for new functionality • Be on the lookout to replace functional tests more fine-grained tests at every given opportunity © 2012 ThoughtWorks, Inc. 19
- 20. What To Automate? • The MOST important scenarios o End-To-End system level scenarios o Happy paths • Avoid automating individual feature- level acceptance criteria • Make a distinction between “ephemeral” and “long-standing” tests © 2012 ThoughtWorks, Inc. 20
- 21. When do these tests run? • After every check-in • As part of CI build pipeline o Unit à Smoke à Regression à Deploy • Incubator Test Suites © 2012 ThoughtWorks, Inc. 21
- 22. Workflow • Sprint [N -1] o Feature elaboration with entry level acceptance criteria • Sprint [N] o Feature kick-off meeting o Developers implement feature o Testers (optionally) add additional acceptance criteria • Sprint[N, N + 1] o Testers verify all acceptance criteria to be satisfied o Testers refactor functional test suite o Feature marked complete © 2012 ThoughtWorks, Inc. 22
- 23. Technology © 2012 ThoughtWorks, Inc. 23
- 24. Test Characteristics • Be idempotent o At least re-runnable o Keep tests isolated from one another • Be runnable in parallel o No inter-dependencies between tests • Be deterministic o Quarantine/eradicate non-deterministic tests o Don’t “sleep” • Prefer callbacks or at least poll for asynchronous responses o Consider mocking remote third-party services • Have separate tests to verify interaction with remote services o Consider using relative dates/times or mock the system clock o http://martinfowler.com/articles/nonDeterminism.html © 2012 ThoughtWorks, Inc. 24
- 25. Test Sources • Don’t have any dependencies on production sources • Treat as first-class citizens o Keep DRY o Run static analysis metrics • Leverage design patterns like o Page Objects o Compound Steps • Consider the use of terser languages o Groovy, Ruby etc. © 2012 ThoughtWorks, Inc. 25
- 26. Test Data • Use application to create test data • Use test data creation steps o Frameworks like DBUnit • Use anonymized production data subsets o Requires lot of time and effort • Use anonymized production data copies o Huge data volumes may make it prohibitive • Avoid the shared database anti-pattern • Consider mocking remote third-party services © 2012 ThoughtWorks, Inc. 26
- 27. Test Infrastructure • Invest in fast build hardware o CI servers support multiple remote agents • Use same software configuration as production in all environments • Consider running tests on the cloud o http://www.saucelabs.com © 2012 ThoughtWorks, Inc. 27
- 28. People © 2012 ThoughtWorks, Inc. 28
- 29. Who Writes/Owns The Tests? • The specifications – acceptance criteria? o Domain Experts • Step implementations? o Developers & Testers • The functional suite? o The Team o Day-to-Day – Testers © 2012 ThoughtWorks, Inc. 29
- 30. Team Dynamics • Fix broken builds and tests immediately o OK to occasionally break the build o Not OK to leave the build broken for long • Do not allow check-ins over a broken build o Unless check-in is being made to fix the build • Co-locate teams as much as possible o At least create fully formed remote teams © 2012 ThoughtWorks, Inc. 30
- 31. Questions? premanand@thoughtworks.com Thank You! © 2012 ThoughtWorks, Inc. 31