Nowadays in IT market, most of enterprises are trying to adopt agile and DevOps methodology to meet the time-to-market expectations and continue satisfying their customers. As the result, continue deploying new applications are become challenges for any software development teams. During the development cycles, several questions has been identified and one of the most interesting questions is How to fit automated tests into agile projects because within agile sprints, there is simply not enough time to automate the set of tests? Action Based Testing (ABT) methodology is becoming a solution to help you achieve your expectations on automated test coverage within the Agile iterations/sprints. ABT uses a modular keyword-driven approach which tests are organized in test modules and are built up of sequences of actions. Well-defined test modules can provide a healthy framework for teams to work with, in particular if modules have a clear and unambiguous scope, the scope is well-differentiated from other test modules, and all test cases …within the test module reflect the scope. A key differentiation is between business tests and interaction tests. Business tests have a business-oriented scope and should not contain UI details. Interaction tests focus on the interaction between the user (or another system) and the application. This topic is about how to apply ABT methodologies into SDLC with some discussions on the three Holy Grail of test design approaches from Hans Buwalda.

1. Ngo Hoang Minh
TEST DESIGN WITH ACTION-
BASED TESTING
METHODOLOGY

2. Who is your speaker?
2
• Software testing company, around since 1994
• Testing and test automation services:
– consultancy, training
– test development and automation services
– "test integrated" development services
• Products:
– TestArchitect™, TestArchitect for Visual Studio™
– integrating test development with test management and automation
– based on modularized keyword-driven testing
• LogiGear Magazine:
– themed issues, non-commercial
LogiGear Corporation
 Background in computer science, management
 Since 2009 focusing on automated testing
 keywords, agile testing
Ngo Hoang Minh

3. 3
Test Design With Action-Based

4. 4
• Introduction
• Action Based Testing, Test Modules
• Success factors of test design (for automation)
• Test case design
• Action design
• Final
Agenda

5. 5
Introduction

6. Testing Under Pressure
6
specification development test
DEADLINE

7. Testing Under Pressure –
"Shift Left"
7
specification development test
DEADLINE
Develop tests in time:
• Test design
• Auditing, acceptance
• Preparations
• Automation

8. 8
Action Based Testing

9. Risks of keyword and scenario approaches
9
• Often seen as silver bullet, complications are
underestimated
– often treated as a technical "trick"
– testers can get squeezed and marginalized
• developers and users dictating tests
• automation engineers dictating actions
– testers can end up with an automation responsibility, thus becoming pseudo
programmers
• The method needs understanding and experience to be
successful
– pitfalls are many, and can have a negative effect on the outcome
• Lack of method and structure can risk manageability
– maintainability not as good as hoped
– results can be disappointing, approach will be blamed

10. Action Based Testing (ABT)
10
• Uses the keyword based "actions" as basis for a method
– Covers test management, test development and automation
– With a large focus on test design as the main driver for automation
success
– Method is specific, but concepts are generic
• The central product in ABT is the "Test Module", not the test
case
– Like chapters in a book
• test cases are part of the test modules,
– Test development is seen as having both analytical and creative aspects
– Developed as spread sheets, external from the automation
– Each test module is a separate (mini) project, each test module can
involve different stake holders

11. Test Module
11
Test Module
Initial - setup
Objectives
Test cases
Final - cleanup

12. Test Modules versus Test Cases
12
• The test module is a bigger unit in the test design
– Easier to identify
– A chapter rather than a paragraph
• Better flow of execution
– Each test case can set up for the next one
– Keep test modules independent, test cases can be dependent
• Test cases become creative output, rather than stifling input
– Avoids having to define all test cases at once early in the process
• Clear scope helps to identify cases, actions and checks

13. 13
Test Design Success Factors

14. Why Better Test Design?
14
• Better test design can improve quality of test
– Many tests are often quite "mechanical" now, no surprises
– One to one related to specifications, user stories or requirements,
which often is ok, but lacks aggression
– No combinations, no unexpected situations, lame and boring
– Such tests have a hard time finding (interesting) bugs
• Better test design can give (much) better automation
– Unneeded details are left out of tests
– Avoiding "over checking“
– Limit the impact of system changes on tests, making such impact
more manageable

15. The Three “Holy Grails” of Test
Design
15
• Metaphor to depict three main steps in test design
• Using "grail" to illustrate that there is no single perfect
solution, but that it matters to pay attention
Organization of tests into test modules
Right approach for each test module
Proper level of detail in the test specification

16. What's the trick...
16

17. What's the trick...
17
• Have or acquire facilities to store and organize your
content
• Select your stuff
• Decide where to put what
– assign and label the shelves
• Put it there
• If the organization is not sufficient anymore, add to it
or change it

18. Properties of a good Breakdown
18
 Reflects the level of tests
 Well differentiated and clear in scope
 Balanced in size and amount
 Modules mutually independent
 Fitting the priorities and planning of the project

19. Breakdown Criteria
19
• Common Criteria
– Functionality (customers, finances, management information, UI, ...)
– Architecture of the system under test (client, server, protocol, sub systems,
components, modules, ...)
– Kind of test (navigation flow, negative tests, response time, ...)
• Additional Criteria
– Stakeholders (like "Accounting", "Compliance", "HR", ...)
– Complexity of the test (put complex tests in separate modules)
– Execution aspects (special hardware, multi-station, ...)
– Project planning (availability of information, timelines, sprints, ...)
– Risks involved (extra test modules for high risk areas)
– Ambition level (smoke test, regression, aggressive, …)

20. Approach 1: Workshop
20
• Gather a meeting with relevant participants
– test developers
– domain experts
– automation engineer (focus on efficiency of automation)
– experienced moderator
– also consider: developers, managers
• If necessary, provide training of participants
before the discussion
• I prefer this approach, in particular to start off

21. Approach 2: Design and Feedback
21
• One or two experienced test designers create a
first draft
• The draft is delivered/discussed to relevant
parties
• Ask the parties to verify:
1. Structure: does it make sense
2. Completeness: are all relevant areas covered
• Based on feedback, further modify the design

22. 22
Test Case Design

23. Grail 2: Approach per Test Module
23
 When to develop: do we have enough information?
 When to execute: make sure lower level stuff working first
Plan the test module:
 Do an intake: understand what is needed and devise an approach
 Analyze requirements, formulate "test objectives", create tests
Process :
 See the test development as a "learning process", about the business domain, the
application structure, the interaction, etc
 Talk about your tests, make them strong
Exploratory approach:
 Users, subject matter experts, developers, auditors, etc
Identify stakeholders and their involvement:
 Users, subject matter experts, developers, auditors, etc
Identify stakeholders and their involvement:

24. Clarify your objectives with your tests . . .
24
requirement,
specification,
…
test case
requirement,
specification,
…
test objective test case
direct relation indirect relation via a test objective
Linking through test objectives can help easier traceability:
...
TO-3.51 The exit date must be after the entry date
...
test objective TO-3.51
name entry date exit date
enter employment Bill Goodfellow 2016-10-02 2016-10-01
check error message The exit date must be after the entry date.

25. Test Objectives
25
• Keep test objectives short and simple
• Focus on what to test, not how
• Split longer texts into atomic sentences
• Typically test objectives will be like:
– cause and effect ("clicking clear clears all fields")
– condition and effect ("if all fields filled, 'ok' is enabled")

26. Eye on the ball, Scope
26
• Always know the scope of the test module
• The scope should be unambiguous
• The scope determines many things:
– what the test objectives are
– which test cases to expect
– what level of actions to use
– what the checks are about and which events should
generate a warning or error (if a “lower” functionality is
wrong)

27. Use of "Anti-patterns"
27
• Informal way to classify typical test design problems
• Use with care, can come across offensive
• The point of view in the following list is automation: test
design choices that can be counter-productive to
automation
• However, lack of good organization can also effect the
quality of manual tests

28. 28
Action Design

29. Grail 3: Specification Level, choosing
actions
29
• Scope of the test determines the action level
• As high level as appropriate, as little arguments as possible
• Clear names for actions
• Avoid "engineer" styles for names of actions and arguments
– tests are not source code
– like no spaces, uppercase, camel-case or underlines
• Manage and document the Actions
• Try to treat as a "by-product of test design"

30. Low-level, high-level, mid-level
actions
30
enter customer
enter address fields
enter select set . . .. . .
"Low level": detailed interaction with the UI (or API)
- generic, do not show any functional or business logic
- examples: "click", "expand tree node", "select menu"
"Mid level": common sequences at a more
detailed application level
- usually to wrap a form or dialog
- for use in high level actions
- greatly enhance maintainability
- example: "enter address fields"
"High level": a business domain operation or check on the
application under test
hide the interaction
examples: "enter customer", "rent car", "check balance"

31. Use the right level actions
31
Low level of UI interaction detail makes sense only with the module
scope is to test the UI
window tree tree item path
click tree item main projects /Projects/Drill Assembly
window list item
check list item exists main tasks Plan of Approach
Better to have a business level action and hide the details in an action
definition:
But do show details when it matters. The example below is too high level,
requires drill down into action definition to understand what is tested.

32. 32
Finally

33. Takeaways
33
• Test design is a major contributor to automation success,
often more than technical prowess
• Domain language approaches like Actions and BDD allow
for efficient communication and driving of automation
• Test modules can help organize the tests, and focus their
scopes
• Focusing tests, checks and actions on a clear and
differentiated scope will make for better tests, but also
better automation

34. © 2014 HCMC Software Testing Club
THANK YOU