In the beginning there wasa model: Using requirements models to drive rigorous test automation

© Curiosity Software Ireland Ltd. 2020
In the beginning there was
a model
Using requirements models to drive
rigorous test automation
Moderator: Tom Pryce
Communication Manager, Curiosity
Thomas.Pryce@Curiosity.Software
@TomTestsToo
Jim Hazen
Performance Test Engineer
Switchfly
James Walker
Director of Technology, Curiosity.
James.Walker@Curiosity.Software
@CuriositySoft

In the beginning there was a model
1. Where is the value of models across design, dev and test?
2. How can we realise these benefits in practice?
3. Demo: Model-based design and test automation
4. Q&A and discussion

Modeling Methods for
Automation in Testing
Jim Hazen

What is a Model?
• Merriam-Webster definitions1
• an example for imitation or emulation
• a description or analogy used to help visualize something that cannot be directly
observed
• a computer simulation based on a system
• Basically, a physical representation of a thing or system for emulation and
visualization.

A model

Another model

Systems Modeling
• Systems modeling is the use of models to conceptualize and construct
systems in business and IT development.
• A common type of systems modeling is function modeling, with specific techniques
such as Functional Flow Diagram’s (FFD).
• Another type of systems modeling is architectural modeling which uses the
systems architecture to conceptually model the structure and behavior of a
system.

A Systems Model

Modeling Techniques
• Traditional Modeling Techniques
• Unified Modeling Language (UML), Function Flow Diagram, Data Flow
Diagram, State Transition diagram, Architectural diagram
• After initial specification, not fully collaborative
• Modern Modeling Techniques
• Story Boarding/Mapping3, Mind Map4, User Community Modeling Language
(UCML)5, Visual Task Analysis6, Behavior Based Testing (BBT) Map7
• Can be part of initial specification, collaborative

Modeling Tests & Scenarios
• Models are a visual representation of a system
• Allow for determining different paths in system
• Allow for determining the levels of functionality
• Allow for defining the data or different states of data
• Allow for identifying the other systems to interact with
• Can be used to design tests
• Can be used to determine test scenarios
• Aid in the development of automation code (types of functionality needed in
framework)

Types of Models -
Traditional

Modeling – Functional Flow Diagram
(FFD)

Application to Automation
• Provides information about the functionality present
• Provides information about logic flow of functionality
• Provides visual representation of paths through functionality
• Provides information about “users” of functionality
• Helps with design of automation code functionality needed in framework
• Helps with automation code modularity design

Modeling – Data Flow Diagram (DFD)

• Provides information about the data used/processed by the system under test
• Provides information about transition states of data as it moves through the system
• Provides visual representation of data relationships as it moves through the system
• Provides information about functionality used to process data at various points in system
• Helps with design of Test Data

Modeling – State Transition Diagram

• Finite State Machine (FSM)
• Provides information about states and transition as “user” moves through
the system, and the actions that cause them
• Provides visual representation of paths through the system, both primary
and alternative ones
• Helps with design of Test Scenarios

Thoughts On Traditional Methods
Pro’s Con’s
Standardized methods and formats Can be rigid in process
High level graphical representation Decomposing into more detail is problematic
Each method is supported by many tools Cost of tools can be prohibitive
Diagrams give information of process Methods are specific to design and not testing,
have to interpret into tests
Different methods & diagrams give different
pieces of information

Types of Models - Modern

Always
Used
Seldom
Used
Criticality
(UsageLevel)
Usage Sequence
Log On
Customer
Management
Order
Processing
Accounting
Inventory
Management
Add New
Customer
New Order
Accounts
Receivable
Inventory
Report
Edit Customer Modify Order Add Inventory
Delete
Customer
Delete Order
Accounts
Payable
Modeling – Story Mapping

• Provides information about the functionality present
• Provides information about usage levels of functionality
• Provides visual representation of paths through functionality
• Helps with design of automation code functionality needed in framework
• Helps with automation code modularity design
• Helps with prioritization of automation code development

Mail
Application
Compose
Can user send mail
Can user send mail with blank subject
Can user send mail with blank body
Can user send mail to Cc, Bcc list recipients
Add Attachment
Single Attachment
Multiple Attachments
Different File Formats
File greater than 25 MB
Insert HTML Link
Insert File
Modeling – Mind Map

• Provides information about the functionality
• Provides information about usage of functionality
• Provides visual representation of paths of functionality
• Provides visual representation of data
• More atomic/granular view of functionality to test
• Helps with design of test scenarios and combinations
• Helps with prioritization of automation code development

Modeling - UCML

• Provides information about the usage level of functionality
• Provides visual representation of paths of usage
• Helps with design of test scenarios and combinations, including End-to-End
• Helps with determining workload model for Performance Testing
• Helps with defining Non-Functional Requirements

UI
JS
API
APP
DB
Successful Login Flow
Load
Page
Potential
Check/Test
Seam /
Layer
Action
Build
Login
Page
Interact
with
Login
Send
API
Request
Process
API
Request
Check
Credentials /
Create token
Send API
Response
Flag as
Logged In
Render
Home page
Home
page
Modeling – Visual Task Analysis

• Provides information about Technology Stack
• Provides a visual representation of topology of functionality layers/seams
• Helps with determining what to interact with and which tool may be utilized
• Helps to decide the types of data needed to drive the tests
• Start testing pieces of the flow in isolation

Modeling – Behavior Based Testing

• Provides information about the data needed for test
• Helps with design of test scenario
• Helps with defining generic data needed for test
• Helps with defining Gherkin’s “Given, When, Then” structure for associated input
data, action, and end state condition or validation

Modeling –Business Process Model
and Notation (BPMN)

• Information about the functionality
• Information about usage of functionality
• Information about the data needed for test
• Helps w/ design & generation of test scenarios & data
• Can be used to generate code for automation in testing
• Helps with design of test scenario
• Helps with defining generic data needed for test

Thoughts On Modern Methods
Pro’s Con’s
Multiple methods and formats Picking the “right” one(s) can be problematic
High level graphical representation, and can have
specific details
Can get to be too detailed, lose the forest for the
trees.
Lots of tools Picking the “right” one can be problematic
Diagrams give information of process Different methods & diagrams give different
pieces of information
Some methods & diagrams are geared towards
testing

References
1. “Model”, Miriam-Webster Dictionary, www.merriam-webster.com/dictionary/model
2. “The New Model for Testing”, Paul Gerrard, Agile Testing Days, https://youtu.be/1Ra1192OpqY, 2017.
3. "Story Mapping," Agile Alliance, www.agilealliance.org/glossary/storymap/, 2018.
4. “What is a Mind Map? - Definition & Examples”, Study.com, study.com/academy/lesson/what-is-a-mind-map-
definition-examples.html, 2018.
5. “User Community Modeling Language (UCMLTM)," Scott Barber, www.perftestplus.com/articles/ucml.pdf, 2003.
6. “Pyramids Are Ancient - Let’s Talk Automation Strategy”, Richard Bradshaw, SauceCon 2019 presentation,
https://youtu.be/83aHKC5C14Q, 2019.
7. “Tired of Your Software Development Testing Automation Failing Over Time?”, Tim Harrison, SQA2,
http://sqasquared.com/blog/2019/07/23/tired-software-development-testing-automation-failing-time/, 2019.

How can we realise these
benefits in practice?
An approach for realising as many model-based benefits as possible

Problems worth solving
Design 64% of bugs emerge in the
requirements analysis and
design phase.1
Requirements defects can account
for as much as 82% of total defect
remediation efforts.2
A bug found in testing costs 15 times more
to fix than one found during design.3
Dev Developers spend 50.1% their time fixing bugs.4 50% of developers encounter the most
delays during testing.5
QA/Testing Testing still consumes
23% of the average IT
budget.6
Fixing a bug found post- release
costs double fixing one found in
QA.3
Only 19% of
UATs today are
automated.6
44% of QA time is spent
searching, managing and
generating data.7
[1] P Mohan, A Udaya Shankar, K JayaSriDevi (2012), “Quality Flaws: Issues and Challenges in Software Development”, in Computer Engineering and Intelligent Systems (3:12:40-48).
[2] James Martin, cited from Bender RBT (2009), Requirements Based Testing Process Overview.
[3] IBM, cited in M Dawson et al (2010), “Integrating Software Assurance into the Software Development Life Cycle (SDLC)”, in Journal of Information Systems Technology & Planning, (3:6: 49-43).
[4] T Britton et al (2013), "Reversible Debugging Software", report created for the Judge Business School, University of Cambridge.
[5] GitLab (2019), 2019 Global Developer Report. [6] Capgemini, Sogeti, Micro Focus (2019), World Quality Report 2019-20. [7] Capgemini, Sogeti, Broadcom (2020), Continuous Testing Report 2020.

Silo Wall Information Hop Delay
No feedback
or traceability
Dev Automation EngineersQA/TestBAs
Effort: I write up lots
of repetitive user stories,
requests and tickets.
Quality:Most bugs
originate in incomplete,
ambiguous requirements.
Effort: I recode after
misunderstood designs
and uncaught bugs.
Quality:Bugs are more
damaging the later they
are discovered.
Effort: I slowly create
and maintain 1000s of
low-coverage tests.
Quality:Manually
derived tests hit a
fraction of system logic.
Effort: I maintain a
growing mountain of
repetitive test scripts.
Quality: Data clashes,
false positives, false
negatives, low coverage.
Formulate complex requirements
and change requests in
disparate, unconnected formats.
Convert the incomplete designs
into code, trying to fix the impact
of changes on complex systems.
Unsystematically create high
volume, low coverage test cases,
mismatched with system designs.
Choose between creating new
scripts for new logic or
maintaining existing code.
Flat RQMTs = Silos = effort and errors
Mini-waterfalls – manual effort and mistranslation mounts with each “information hop”

Flowchart modelling
Recorded
tests activity
Test Cases
BPMN
diagrams
BDD
Scenarios
Business
Requirements
Application Scans
Reactive Model
of the system
Find / Make / Allocate Test Data
Run Results
Tests
New
Iteration
 Perfect sets of tests
 Fast Maintenance
 Reusable Components
 Move faster
 Deliver with confidence
Test AutomationTest Automation

Modelling solves automation challenges
1. Modelling automates repetitive test case and data design.
2. Modelling minimises maintenance time – updating models updates all artefacts
generated from them.
3. Test optimisation finds bugs earlier and at less cost to fix.
4. Re-usability allows engineers to focus on feeding in new code which is re-used across
cross-functional teams.
5. Modelling minimises test data frustrations, creating unique combinations for every test.

Models are a collaborate
piece
Most bugs stem from requirements.
The best way to test these bugs is to catch them as they arise.
• Collaboratively modelling provides continuous feedback – each
tester, dev, BA etc. spots things the others didn’t.
• Modelling eradicates ambiguity as we pick only one option.
• Visual models make it easy to spot incompleteness upfront.
We then execute tests directly from the Living Documentation,
reducing manual effort and driving quality.
Behaviour-Driven
Scenarios
ChatOps and Oral
Agreements
“Flat” diagrams
Requirements
Documents
Email
Requests
Spoken Agreements
Requirements are often spread over
unconnected formats.
Modelling consolidates them, spotting
incompleteness & ambiguity.
This “living documentation” avoids
technical debt and keeps teams aligned.
⨯

Models pool knowledge and skills
The models then automate the creation test assets and specifications, driving
accurate development and in-sprint testing
Skills sharing & cross-team collaboration:
People who understand the user
requirements
People who understand testing
People who understand automation
People who understand data
People who understand integration
People who understand CI/CD
A single source of truth for all assets
Test scripts
Test Cases
Written Requirements & Diagrams
Gherkin specifications
Test Data
Service Definitions & Virtual Services
Cross-team alignment & real-time communication • No technical debt • In-sprint
automation • Minimal dev rework • Avoid duplicate effect and cumbersome maintenance

Modelling is an accelerator, not
replacement, for good test automation
With modelling, automation becomes a largely automated comparison
of the modelled system to the system built in code.
That means accurate regression tests executed in-sprint.
Feeding in new code, it becomes re-usable in future sprints and by non-
coders.
You can focus on testing what’s new and interesting, testing fast-
changing systems.

Demo

Want to learn more from
Jim?
Check out Before the Code. 15 chapters
of expert guidance on what you should
do before you write a single line of
automation code.
Get your (discounted) copy at
https://bit.ly/3lKqBiI

The broken promise of test automation
Daniel Howard
Senior Researcher
Bloor Research
November 10th 2020 2pm GMT / 9am EST
Huw Price
Managing Director
Curiosity Software
SIGN UP TODAY
James Walker
Director of Technology
Curiosity Software
Why are we still hand-cranking tests?

Carry on the conversation?
• We’d love to chat further. Book a web meeting at
calendly.com/curiosity-james
• Want to try MBT? Visit Testmodeller.io for a free trial.
• Follow @CuriositySoft
• Email us on James.Walker@Curiosity.Software
• Connect with Jim and James on LinkedIn

In the beginning there wasa model: Using requirements models to drive rigorous test automation

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a In the beginning there wasa model: Using requirements models to drive rigorous test automation

Semelhante a In the beginning there wasa model: Using requirements models to drive rigorous test automation (20)

Mais de Curiosity Software Ireland

Mais de Curiosity Software Ireland (17)

Último

Último (20)