Managing Successful Test Automation

Managing Successful Test Automation
Contents
Session 0: Introduction to the tutorial
Objectives, What we cover (and don’t cover) today
Session 1: Planning and Managing Test Automation
Responsibilities
Pilot project
Test automation objectives (and exercise)
Measures for automation
Return on Investment (ROI) (and exercise)
Session 2: Testware Architecture
Importance of a testware architecture
What needs to be organised
Session 3: Pre- and Post-Processing
Automating more than tests
Test status
Session 4: Scripting Techniques
Objectives of scripting techniques
Different types of scripts
Domain specific test language
Session 5: Automated Comparison
Automated test verification
Test sensitivity
Comparison example
Session 6: Final Advice, Q&A and Direction
Strategy exercise
Final advice
Questions and Answers
Appendix (useful stuff)
That’s no reason to automate (Better Software article)
Effective Automated Testing with a DSTL, Martin Gijsen
Man and Machine, Jonathan Kohl (Better Software)
Technical vs non-technical skills in test automation

0-1

Managing
Successful Test Automation
Prepared and presented by

Dorothy Graham
www.DorothyGraham.co.uk
email: info@dorothygraham.co.uk
Twitter: @DorothyGraham
© Dorothy Graham 2013
0-1

Objectives of this tutorial
•  help you achieve better success in automation
–  independent of any particular tool

•  mainly management and some technical issues
–  objectives for automation
–  showing Return on Investment (ROI)
–  importance of testware architecture
–  practical tips for a few technical issues
–  what works in practice (case studies)

•  help you plan an effective automation strategy
0-2

presented by Dorothy Graham
info@dorothygraham.co.uk


0-2

Tutorial contents
1) planning & managing test automation
2) testware architecture
3) pre and post processing
4) scripting techniques
5) automated comparison
6) final advice, Q&A and direction

0-3

Shameless commercial plug

Part 1: How to do
automation - still relevant
today, though we plan to
update it at some point

New
book!

0-4



0-3

What is today about? (and not about)
•  test execution automation (not other tools)
•  I will NOT cover:
–  demos of tools (time, which one, expo)
–  comparative tool info (expo, web)
–  selecting a tool*

•  at the end of the day
–  understand technical and non-technical issues
–  have your own automation objectives
–  plan your own automation strategy
* I will email you Ch 10 of the STA book on request – info@dorothygraham.co.uk

0-5

Test automation survey 2004 - 2010
–  survey by Erik van Veenendaal in Professional
Tester magazine (Nov/Dec 2010)
– 
–  have test execution tools
–  have shelfware tools

2004
29%
26%

2010
48%
28%

•  most common shelfware – test execution tools (33%)

–  achieving many benefits

27%

50%

0-6



0-4

Current tool adoption (Australia)
•  K.J. Ross & Associates survey 2010
–  19%
–  7%

good benefits
using tools but not anticipated ROI

26% success
(partial)
74% failure!

–  35%
–  14%
–  12%
–  10%
–  3%

don’t have skills to adopt or maintain
automation just an added burden, no time
tools abandoned – too much maintenance
tools bought, no resources to adopt
no budget to buy or adopt tools

Source: www,kjross.com.au (good webinar on automation)

0-7

About you
•  your Summary and Strategy document
–  where are you now with your automation?
–  what are your most pressing automation
problems?
–  why are you here today?
There is small group work throughout the day (maximum of 5 per group);
introduce yourselves within your group.
0-8



1-Managing


Planning & Managing Test Automation

1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice

1-1

Managing


1

2

3

4

5

6

Contents
Responsibilities
Pilot project
Test automation objectives
Return on Investment (ROI)
1-2



1-Managing

What is an automated test?
•  a test!
–  designed by a tester for a purpose

•  test is executed
–  implemented / constructed to run automatically
using a tool
–  could be run manually also

•  who decides what tests to run?
•  who decides how a test is run?
1-3

Existing perceptions of automation skills
•  many books & articles don’t mention
automation skills
–  or assume that they must be acquired by testers

•  test automation is technical in some ways
•  using the test execution tool directly (script writing)
•  designing the testware architecture (framework /
regime)
•  debugging automation problems

–  this work requires technical skill
–  most people now realise this (but many still don’t)
See article: “Technical vs non-technical skills in test automation”


1-4


1-Managing

Responsibilities
Testers
•  test the software
–  design tests
–  select tests for automation
•  requires planning / negotiation

Automators
•  automate tests (requested by
testers)
•  support automated testing
–  allow testers to execute tests
–  help testers debug failed tests
–  provide additional tools (homegrown)

•  execute automated tests
–  should not need detailed
technical expertise

•  analyse failed automated
tests
–  report bugs found by tests
–  problems with the tests may
need help from the automation
team

•  predict
–  maintenance effort for software
changes
–  cost of automating new tests

•  improve the automation
–  more benefits, less cost

1-5

Test manager’s dilemma
•  who should undertake automation work
–  not all testers can automate (well)
–  not all testers want to automate
–  not all automators want to test!

•  conflict of responsibilities
–  automate tests vs. run tests manually

•  get additional resources as automators?
–  contractors? borrow a developer? tool vendor?
1-6



1-Managing

Recent poll
Testers as automators?
Every tester should be able to
write code
Testers can be automators if
they want to
Testers shouldn't have to
become automators
Testers should never be
automators
0 10 20 30 40 50 60 70 80
EuroStar Webinar, Intelligent Mistakes in Test Automation, 20 Sept 2012

1-7

Roles within the automation team
•  Testware architect
–  designs the overall structure for the automation

•  Champion
–  “sells” automation to managers and testers

•  Tool specialist / toolsmith
–  technical aspects, licensing, updates to the tool

•  Automated test (& script) developers
–  write new keyword scripts as needed
–  debug automation problems
1-8



1-Managing

Agile automation: Lisa Crispin
–  starting point: buggy code, new functionality
needed, whole team regression tests manually
–  testable architecture: (open source tools)
•  want unit tests automated (TDD), start with new code
•  start with GUI smoke tests - regression
•  business logic in middle level with FitNesse

–  100% regression tests automated in one year
•  selected set of smoke tests for coverage of stories

–  every 6 mos, engineering sprint on the automation
–  key success factors
•  management support & communication
•  whole team approach, celebration & refactoring
1-9

Chapter 1, pp 17-32, Experiences of Test Automation

Automation and agile
•  agile automation: apply agile principles to
automation
–  multidisciplinary team
–  automation sprints
–  refactor when needed

•  fitting automation into agile development
–  ideal: automation is part of “done” for each sprint
•  Test-Driven Design = write and automate tests first

–  alternative: automation in the following sprint ->
•  may be better for system level tests
See www.satisfice.com/articles/agileauto-paper.pdf (James Bach)


1-10


1-Managing

Automation in agile/iterative development
A

manual testing of
this release (testers)
A

B

regression testing (automators automate the best tests)

A

B

C

run automated tests (testers)
A

B

C

D

E

F
1-11

Requirements for agile test framework
•  Support manual and automated testing
–  using the same test construction process

•  Support fully manual execution at any time
–  requires good naming convention for components

•  Support manual + automated execution
–  so test can be used before it is 100% automated

•  Implement reusable objects
•  Allow “stubbing” objects before GUI available
Source: Dave Martin, LDSChurch.org, email


1-12


1-Managing

Managing


1

2

3

4

5

6

Contents
Responsibilities
Pilot project
1-13

A tale of two projects: Ane Clausen
–  Project 1: 5 people part-time, within test group
•  no objectives, no standards, no experience, unstable
•  after 6 months was closed down

–  Project 2: 3 people full time, 3-month pilot
•  worked on two (easy) insurance products, end to end
•  1st month: learn and plan, 2nd & 3rd months: implement
•  started with simple, stable, positive tests, easy to do
•  close cooperation with business, developers, delivery
•  weekly delivery of automated Business Process Tests

–  after 6 months, automated all insurance products


1-14


1-Managing

Pilot project
•  reasons
–  you’re unique
–  many variables /
unknowns at start

•  benefits
–  find the best way for you
(best practice)
–  solve problems once
–  establish confidence
(based on experience)
–  set realistic targets

•  objectives
–  demonstrate tool value
–  gain experience / skills
in the use of the tool
–  identify changes to
existing test process
–  set internal standards
and conventions
–  refine assessment of
costs and achievable
benefits
1-15

What to explore in the pilot
•  build / implement automated tests (architecture)
–  different ways to build stable tests (e.g. 10 – 20)

•  maintenance
–  different versions of the application
–  reduce maintenance for most likely changes

•  failure analysis
–  support for identifying bugs
–  coping with common bugs affecting many
automated tests
Also: naming conventions, reporting results, measurement


1-16


1-Managing

After the pilot…
•  having processes & standards is only the start
–  30% on new process
–  70% on deployment

Source: Eric Van Veenendaal,
successful test process improvement

•  marketing, training, coaching
•  feedback, focus groups, sharing what’s been done

•  the (psychological) Change Equation
–  change only happens if (x + y + z) > w
x = dissatisfaction with the current state
y = shared vision of the future
z = knowledge of the steps to take to get from here to there
w = psychological / emotional cost to change for this person
1-17

Managing


1

2

3

4

5

6

Contents
Responsibilities
Pilot project
1-18



1-Managing

An automation effort
•  is a project
–  with goals, responsibilities, and monitoring
–  but not just a project – ongoing effort is needed

•  not just one effort – different projects
–  when acquiring a tool – pilot project
–  when anticipated benefits have not materialized
–  different projects at different times
•  with different objectives

•  objectives are important for automation efforts
–  where are we going? are we getting there?

1-19

A manual vs an automated test
an automated
test

Effective

Economic
Economic

first Run of an
automated test

an interactive
(manual) test

Evolvable

Exemplary
1-20



1-Managing

Good objectives for automation?
–  run regression tests evenings and weekends
–  give testers a new skill / enhance their image
–  run tests tedious and error-prone if run manually
–  gain confidence in the system
–  reduce the number of defects found by users
1-21

Objectives Exercise

1-22



Test Automation Objectives Exercise

The following are some possible test automation objectives. Evaluate each objective – is it
a suitable objective for automation? If not, why not?
Which are already in place in your own organisation?

Possible test automation objectives
Achieve faster performance for the
system

Good
automation objective?
(If not, why not)

Already in
place?

NO – this is not an objective for test
execution automation, nor is it an objective for
performance testing! Performance test tools
may help by giving the measurements to see
whether the system is faster.

Achieve good results and quick
payback with no additional resources,
effort or time
Automate all tests
Build a long-lasting automation regime
that is easy to maintain
Easy to add new automated tests
Ensure repeatability of regression tests
Ensure that we meet our release
deadlines
Find more bugs
Find defects in less time
Free testers from repeated (boring) test
execution to spend more time in test
design

© Dorothy Graham

STA1110126

Page 1 of 5



Good
(If not, why not)

Already in
place?

Improve our testing
Reduce elapsed time for testing by x%
Reduce the cost and time for test
design
Reduce the number of test staff
Run more tests
Run regression tests more often
Run tests every night on all PCs
Achieve a positive Return on
Investment in no more than <x> test
interations
(where x = ?)
Other objectives:

© Dorothy Graham

STA1110126

Page 2 of 5

1-Managing

Reduce test execution time
edit tests
(maintenance) set-up

execute

analyse
failures

clear-up

Manual
testing

Same tests
automated

More mature
automation

1-23

Automate x% of the tests
•  are your existing tests worth automating?
–  if testing is in chaos, automating gives you faster
chaos

•  which tests to automate (first)?
•  what % of manual tests should be automated?
–  “100%” sounds impressive but may not be wise

•  what else can be automated
–  automation can do things not possible or practical
in manual testing!
1-24



1-Managing

Manual vs automated
manual
tests

automated
tests

tests not
automated
yet

tests not
worth
automating

tests (&
verification)
not possible to
do manually
manual tests
automated
(% manual)

exploratory
test
automation
1-25

Exploratory Test Automation - 1
•  sounds like an oxymoron?
•  when you are exploring, you might say
–  “that’s weird – might there be any more like that?”

•  is there a way to check a result automatically?
–  a heuristic oracle

•  can you generate lots of different (random)
inputs?
•  have you got lots of computer power available?
Source: Harry Robinson tutorial at CAST 2010, August 2010


1-26


1-Managing

Exploratory Test Automation - 2
•  set off lots of tests that produce something that
can be checked with an automated oracle
•  alerts a human when something unusual occurs
•  it’s exploratory – we don’t know what we will
find, we don’t have a planned route through the
system
•  it’s automated – a “shotgun” approach – lots of
different (random) inputs
•  can find a class of bug too hard to find manually
–  because so many tests can be run

1-27

Success = find lots of bugs?
•  tests find bugs, not automation
•  automation is a mechanism for running tests
•  the bug-finding ability of a test is not affected
by the manner in which it is executed
•  this can be a dangerous objective
–  especially for regression automation!
1-28



1-Managing

What is automated?
most often
automated
likelihood of
finding bugs

regression tests

exploratory testing
1-29

When is “find more bugs” a good
objective for automation?
•  objective is “fewer regression bugs missed”
•  when the first run of a given test is
automated
–  MBT, Exploratory test automation, automated
test design
–  keyword-driven (e.g. users populate
spreadsheet)

•  find bugs in parts we wouldn’t have tested?
1-30



1-Managing

Good objectives for test automation
•  measurable
–  EMTE (Equivalent Manual Test Effort)
–  tests run, coverage (e.g. features tested)

• 
• 
• 
• 

realistic and achievable
short and long term
regularly re-visited and revised
should be different objectives for testing and
for automation
•  automation should support testing activities
1-31

Trying to get started: Tessa Benzie
–  consultancy to start automation effort
•  project, needs a champion – hired someone
•  training first, something else next, etc.

–  contract test manager – more consultancy
•  bought a tool – now used by a couple contractors
•  TM moved on, new QA manager has other priorities

–  just wanting to do it isn’t enough
•  needs dedicated effort
•  now have “football teams” of manual testers
Chapter 29, pp 535, Experiences of Test Automation


1-32


1-Managing

Managing


1

2

3

4

5

6

Contents
Responsibilities
Pilot project
1-33

Useful measures
•  a useful measure:
“supports effective analysis and decision making, and
that can be obtained relatively easily.”
Bill Hetzel, “Making Software
Measurement Work”, QED, 1993.

•  easy measures may be more useful even
though less accurate (e.g. car fuel economy)
•  ‘useful’ depends on objectives, i.e. what you
want to know
1-34



1-Managing

Automation measures?
•  aspects of automation
–  number of times tests are run
–  time to run the automated tests (& manual equiv)
–  effort in automation and running automated tests
•  effort to build new automated tests
•  effort to run automated tests (by people)
•  effort to analyse failed automated tests
•  effort to maintain automated tests

–  number of test failures caused by one s/w fault
–  number of automation scripts
1-35

EMTE – what is it?
•  Equivalent Manual Test Effort
–  given a set of automated tests,
–  how much effort would it take
•  IF those tests were run manually

•  note
–  you would not actually run these tests manually
–  EMTE = what you could have tested manually
•  and what you did test automatically

–  used to show test automation benefit
1-36



1-Managing

EMTE – how does it work?
a manual test

Manual testing

Automate the manual testing?
the manual test
now automated
doesn’t make sense –
can run them more

only time to run the tests
1.5 times

1-37

EMTE – how does it work? (2)
Automated testing

EMTE
1-38



1-Managing

EMTE example
•  example
–  automated tests take 2 hours
–  if those same tests were run manually, 4 days

•  frequency
–  automated tests run every day for 2 weeks
(including once at the weekend), 11 times

•  calculation
–  EMTE =

1-39

Examples of good (measurable)
objectives for test automation
•  achieve positive ROI in less than 6 test iterations
–  measured by comparing to manual testing

•  run most important tests using spare resource
–  top 10% of usefulness rating, run out of hours

•  reduce elapsed time of tool-supported activities
–  measured for maintenance & failure analysis time

•  improve automation support for testers
–  testers rate usefulness of automation support
–  how often utilities/automation features are used
1-40



1-Managing

Managing


1

2

3

4

5

6

Contents
Responsibilities
Pilot project
1-41

Why measure automation ROI?
•  to justify and confirm starting automation
–  business case for purchase/investment decision,
to confirm ROI has been achieved e.g. after pilot
–  both compare manual vs automated testing

•  to monitor on-going automation
–  for increased efficiency, continuous improvement
–  build time, maintenance time, failure analysis time,
refactoring time
•  on-going costs – what are the benefits?
1-42



1-Managing

What can you show as a benefit?
•  number of additional times tests are run
–  how many times run manually & automated

•  how long tests take to run
–  execution time for manual and automated, EMTE

•  how much effort to run tests
–  human effort for manual testing and kicking off/
dealing with automated tests

•  data variety and/or coverage
–  different sets of data run manually & automated
–  different parts of the system now tested

1-43

An example comparative benefits chart
80
70
60
50
40

man
aut

30
20
10
0

exec speed
14 x faster

times run

data variety tester work

5 x more often 4 x more data 12 x less effort

ROI spreadsheet – email me for a copy
1-44



1-Managing

EMTE over releases
EMTE per release
800
700
600
500
400
300
200
100
0
1

2

3

4

5

2708 Total “manual” hours of testing done automatically
339 Days of testing performed

1-45

•  ROI = (benefit – cost) / cost
–  Investment: costs can be expressed as effort
•  which can be converted to money

–  Return: benefits
•  reduced tester time effort
•  but what about things like “faster execution”, “run more
often”, “greater coverage” “quicker time to market”?
•  how to convert these to money?

–  possible calculation based on tester time – the
other things are bonuses?
•  comparing to manual testing
1-46



1-Managing

ROI example
Human Time
to run
Savings per
automated
automated No. automated Total
run
runs per release savings
tests

Manual
time
testing

66

7

Average
Build time
for new aut Maintenance
time
tests

15

59

9

531

Failure
Other
Total
ROI =
analysis automation automation (Gain time
time spent investment Cost) / Cost

50

35

22

122

3.4

Free ROI spreadsheet available
1-47

MBT @ ESA:
Stefan Mohacsi, Armin Beer
–  home-grown tool interfaced to commercial tools
•  Model-Based Testing and Test Case Generation
•  layers of abstraction for maintainability

–  define model before software is ready
•  capture and assign GUI objects later
•  developers build in testability

–  ROI calculations
•  invest 460 hours in automation infrastructure
•  break-even after 4 test cycles


1-48


1-Managing

Example ROI graph using MBT
1400

1200

1000

People
Hours

800
Manual hrs
Automated hrs

600

400

200

0
1

2

3

4

5

6

1-49

Source: Stefan Mohacsi & Armin Beer

Database testing: Henri van de Scheur
–  tool developed in-house (now open source)
•  agreed requirements with relevant people up front
•  9 months, 4 developers in Java (right people)
•  good architecture, start with quick wins
–  flexible configuration, good reporting, metrics used to improve

–  results: 2400 times more efficient
•  from: 20 people run 40 tests on 6 platforms in 4 days
•  to: 1 person runs 200 tests on 10 platforms in 1 day
•  quick dev tests, nightly regression, release tests
•  life cycle of automated tests
•  little maintenance, machines used 24x7, better quality


1-50


1-Managing

Large S Africa bank: Michael Snyman
•  was project-based, too late, lessons not learned
–  “our shelves were littered with tools..”

•  2006: automation project, resourced, goals
–  formal automation process

•  ROI after 3 years
–  US$4m on testing project, automation $850K
–  savings $8m, ROI 900%
•  20 testers for 4 weeks to 2 in 1 week

–  automation ROI justified the testing project
•  only initiative that was measured accurately

1-51

Example ROI graph
Savings % vs Tests
100%
50%
0%
0

500

1000

1500

2000

2500

-50%
-100%
-150%
-200%

monthly

weekly

daily

Source: Lars Wahlberg,
Chapter 18 in “Experiences of Test Automation”


1-52


1-Managing

Why is measuring ROI dangerous?
•  focus on what’s easy to measure (tester time)
•  other factors may be much more important
–  time to market, greater coverage, faster testing

•  defining ROI only by tester time may give the
impression that the tools replace the testers
–  this is dangerous
–  tools replace some aspects of some of what
testers do, with increased cost elsewhere
–  we want a net benefit, even if hard to quantify
1-53

How important is ROI?
•  “automation is an enabler for success, not a
cost reduction tool” (Yoram Mizrachi*)
•  many achieve lasting success without
measuring ROI (depends on your context)
–  need to be aware of benefits (and publicize them)

*“Planning a mobile test automation strategy
that works, ATI magazine, July 2012


1-54


1-Managing

Sample ‘starter kit’ for metrics for test
automation (and testing)
•  some measure of benefit
–  e.g. EMTE or coverage

•  average time to automate a test (or set of
related tests)
•  total effort spent on maintaining automated
tests (expressed as an average per test)
•  also measure testing, e.g. Defect Detection
Percentage (DDP) – test effectiveness
–  more info on DDP on my web site & blog
1-55

Recommendations
•  don’t measure everything!
•  choose three or four measures
–  applicable to your most important objectives

•  monitor for a few months
–  see what you learn

•  change measures if they don’t give useful
information
•  be careful with ROI if based on tester time
–  may give impression tools can replace people!
1-56



1-Managing

Managing


1

2

3

4

5

6

Summary: key points
• 
• 
• 
• 
• 

Assign responsibility for automation (and testing)
Use a pilot project to explore the best ways of
doing things
Know your automation objectives
Measure what’s important to you
Show ROI from automation
1-57

Good objectives for automation?
–  run regression tests evenings and weekends
not a good objective, unless they are worthwhile tests!

–  give testers a new skill / enhance their image
not a good objective, could be a useful by-product

–  run tests tedious and error-prone if run manually
good objective

–  gain confidence in the system
an objective for testing, but automated regression tests help achieve it

–  reduce the number of defects found by users
not a good objective for automation, good objective for testing!
1-58



Test Automation Objectives Solution

We have given some ideas as to which objectives are good and why the others are not.

Good
(If not, why not)

Achieve faster performance for the
system

NO – this is not an objective for test execution
automation, nor is it an objective for performance
testing! Performance test tools may help by giving the
measurements to see whether the system is faster.

Achieve good results and quick
payback with no additional resources,
effort or time

NO – this is totally unrealistic – expecting a miracle
with no investment!

Automate all tests

NO – automating ALL tests is not realistic nor
sensible. Automate only those tests that are worth
automating.

Build a long-lasting automation
regime that is easy to maintain

YES – this is an excellent objective for test
automation, and it is measurable.

Easy to add new automated tests

YES. with a good automation regime, it can be easier
to add a new automated test than to run that test
manually.

Ensure repeatability of regression tests YES. The tools will run the same test in the same way
every time.
Ensure that we meet our release
deadlines

NO. Automation may help to run some tests that are
required before release, but there are many more
factors that go into a release decision.

Find more bugs

NO. Automation just runs tests. It is the tests that find
the bugs, whether they are run manually or are
automated.

Find defects in less time

Not really. Some types of defects (regression bugs)
will be found more quickly by automated tests, but it
may actually take longer to analyse the failures found.

Free testers from repeated (boring) test YES. This is a good objective for test execution
execution to spend more time in test
automation.
design

© Dorothy Graham

STA110126

Page 3 of 5


Good
(If not, why not)

Improve our testing

NO. Better testing practices and better use of
techniques will improve testing.

Reduce elapsed time for testing by x% NO. Elapsed time depends on many factors, and not
much on whether tests are automated (see further
explanation in the slides).
Reduce the cost and time for test
design

NO. Test design is independent from automation – the
time spent in design is not affected by how those tests
are executed.

Reduce the number of test staff

NO. You will need more staff to implement the
automation, not less. It can make existing staff more
productive by spending more time on test design.

Run more tests

YES but only long term. Short term, you may actually
run fewer tests because of the effort taken to automate
them.

Run regression tests more often

YES – this is what the test execution tools do best.

Run tests every night on all PCs

NO. It may look impressive, but what tests are being
run? Are they useful? If not, this is a waste of
electricity.

Achieve a positive Return on
Investment in no more than <6> test
interations

YES. This is a good objective, if the number of
iterations is a reasonable number (e.g. 6).

Other objectives:

© Dorothy Graham

STA110126

Page 4 of 5

Test Automation Objectives: Selection and Measurement
On this page, record the test objectives that would be most appropriate for your
organisation (and why), and how you will measure them (what to measure and how to
measure it). I suggest that you include at least one about showing Return on Investment.
If you currently have automation objectives in place in your organisation that are not good
ones, make sure that they are removed and replaced by the better ones below!

Proposed test automation objective
(with justification)

What to measure and how to measure it

Add any comments or thoughts here or on the back of this page.

© Dorothy Graham

STA110126

Page 5 of 5

2-Architecture


Testware Architecture

1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice

Ref. Chapter 5: Testware Architecture
“Software Test Automation”
2-1

Architecture

1

2

3

4

5


6

Contents

2-2



2-Architecture

Testware architecture

testware'architecture'

Testers''
write'tests'(in'DSTL)'

abstraction here:
easier to write
automated tests
-> widely used

High Level Keywords

structured'
testware'

Test
Automator(s)

Structured Scripts

Test'Execu/on'Tool'
runs'scripts'

abstraction here:
easier to maintain,
and change tools
-> long life

2-3

Architecture – abstraction levels
•  most critical factor for success
–  worst: close ties between scripts, tool & tester

•  separate testers’ view from technical aspects
–  so testers don’t need tool knowledge
•  for widespread use of automation
•  scripting techniques address this

•  separate tests from the tool – modular design
–  likely changes confined to one / few module(s)
–  re-use of automation functions
–  for minimal maintenance and long-lived automation
2-4



2-Architecture

Easy way out: use the tool’s architecture
•  tool will have its own way of organising tests
–  where to put things (for the convenience of the
tool!)
–  will “lock you in” to that tool – good for vendors!

•  a better way (gives independence from tools)
–  organise your tests to suit you
–  as part of pre-processing, copy files to where the
tool needs (expects) to find them
–  as part of post-processing, copy back to where
you want things to live
2-5

Tool-specific script ratio
Testers''

Testers''

Not Toolspecific

Tool-specific
scripts

Test'Execu/on'Tool'

High
maintenance
and/or tooldependence

Test'Execu/on'Tool'
2-6



2-Architecture

Learning is incremental: Molly Mahai
–  book learning – knew about investment, not
replace people, don’t automate everything, etc.
–  set up good architecture? books not enough
–  picked something to get started
•  after a while, realised limitations
•  too many projects, library cumbersome

–  re-designed architecture, moved things around
–  didn’t know what we needed till we experienced
the problems for ourselves
•  like trying to educate a teenager
2-7


Architecture

1

2

3

4

5


6

Contents

2-8



2-Architecture

A test for you
•  show me one of your automated tests running
–  how long will it take before it runs?

•  typical problems
–  fails: forgot a file, couldn’t find a called script
–  can’t do it (yet):
•  Joe knows how but he’s out,
•  environment not right,
•  haven’t run in a while,
•  don’t know what files need to be set up for this script

•  why not: run up control script, select test, GO
2-9

Key issues
•  scale
–  the number of scripts, data files, results files,
benchmark files, etc. will be large and growing

•  shared scripts and data
–  efficient automation demands reuse of scripts and
data through sharing, not multiple copies

•  multiple versions
–  as the software changes so too will some tests but
the old tests may still be required

•  multiple environments / platforms
2-10



2-Architecture

Terms

- Testware artefacts
Testware

Test Materials

Test Results
Products

inputs

By-Products

scripts
doc (specifications)
data
env
utilities

logs
actual
results

expected
results

status

differences
differences
summary
summary
2-11

Testware for example test case
open.scp

saveas.scp

Shared Shared
script
Test script: script
- test input
countries.scp

log.txt

Log

countries.dcm

Initial
Document
testspec.txt

Test
Specification


Scribble

Edited
Document

countries2.dcm

Expected
Output

countries2.dcm

Compare

diffs.txt

Differences
2-12


2-Architecture

Testware by type
Testware
Test Materials
countries.scp
open.scp saveas.scp
testdef.txt
countries.dcm

Test Results
Products

By-Products
log.txt
status.txt

countries2.dcm

countries2.dcm

diff.txt

2-13

Benefits of standard approach
•  tools can assume knowledge (architecture)
–  they need less information; are easier to use;
fewer errors will be made

•  can automate many tasks
–  checking (completeness, interdependencies);
documentation (summaries, reports); browsing

•  portability of tests
–  between people, projects, organisations, etc.

•  shorter learning curve
2-14



2-Architecture

Testware Sets
•  Testware Set:
–  a collection of testware artifacts
–  four types:
•  Test Set - one or more test cases
•  Script Set - scripts used by two or more Test Sets
•  Data Set - data files used by two or more Test Sets
•  Utility Set - utilities used by two or more Test Sets

–  good software practice: look for what is
common, and keep it in only one place!
–  Keep your testware DRY!
2-15

Testware library
–  repository of master
versions of all Testware
Sets
–  uncategorised scripts
worse than no scripts
–  Onaral & Turkmen

–  CM is critical
–  “If it takes too long to update
your test library, automation
introduces delay instead of
adding efficiency”
–  Linda Hayes, AST magazine,
Sept 2010


d_ScribbleTypical
d_ScribbleTypical
d_ScribbleVolume
s_Logging
s_ScribbleDocument
s_ScribbleDocument
s_ScribbleDocument
s_ScribbleNavigate
t_ScribbleBreadth
t_ScribbleCheck
t_ScribbleFormat
t_ScribbleList
t_ScribbleList
t_ScribbleList
t_ScribblePrint
t_ScribbleSave
t_ScribbleTextEdit
t_ScribbleTextEdit
u_ScribbleFilters
u_GeneralCompare

v1
v2
v1
v1
v1
v2
v3
v1
v1
v1
v1
v1
v2
v3
v1
v1
v1
v2
v1
v1

Version
numbers

2-16


2-Architecture

Separate test results

Test
suite

Software
under test

Test
results

2-17

Incremental approach: Ursula Friede
–  large insurance application
–  first attempt failed
•  no structure (architecture), data in scripts

–  four phases (unplanned)
•  parameterized (dates, claim numbers, etc)
•  parameters stored in single database for all scripts
•  improved error handling (non-fatal unexpected events)
•  automatic system restart

–  benefits: saved 200 man-days per test cycle
•  €120,000!


2-18


2-Architecture
Architecture

1

2

3

4

5


6

Summary: key points
• 
• 
• 

Structure your automation testware to suit you
Testware comprises many files, etc. which need to
be given a home
Use good software development standards

2-19



3-Pre and Post Processing


Pre- and Post-Processing

1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice

Ref. Chapter 6: Automating Pre- and Post-Processing
3-1

Pre and Post

1

2

3

4

5


6

Contents
Test status

3-2



What is pre- and post-processing?
•  Pre-processing
–  automation of setup tasks necessary to fulfil test
case prerequisites

•  Post-processing
–  automation of post-execution tasks necessary to
complete verification and house-work

•  These terms are useful because:
–  there are lots of tasks, they come in packs, many
are the same, and they can be easily automated
3-3

Automated tests/automated testing
Automated tests
Select / identify test cases to run
Set-up test environment:
•  create test environment
•  load test data
Repeat for each test case:
•  set-up test pre-requisites
•  execute
•  compare results
•  log results
•  analyse test failures
•  report defect(s)
•  clear-up after test case
Clear-up test environment:
•  delete unwanted data
•  save important data
Summarise results
Manual process


Automated testing
Select / identify test cases to run
Set-up test environment:
•  create test environment
•  load test data
Repeat for each test case:
•  set-up test pre-requisites
•  execute
•  compare results
•  log results
•  clear-up after test case
Clear-up test environment:
•  delete unwanted data
•  save important data
Summarise results
Analyse test failures
Report defects
Automated process

3-4


Examples
•  pre-processing
–  copy scripts from a
common script set (e.g.
open, saveas)
–  delete files that shouldn’t
exist when test starts
–  set up data in files
–  copy files to where the
tool expects to find them
–  save normal default file
and rename the test file
to the default (for this
test)

•  post-processing
–  copy results to where the
comparison process
expects to find them
–  delete actual results if
they match expected (or
archive if required)
–  rename a file back to its
normal default

3-5

Outside the box: Jonathan Kohl
–  task automation (throw-away scripts)
•  entering data sets to 2 browsers (verify by watching)
•  install builds, copy test data

–  support manual exploratory testing
–  testing under the GUI to the database (“side door”)
–  don’t believe everything you see
•  1000s of automated tests pass too quickly
•  monitoring tools to see what was happening
•  “if there’s no error message, it must be ok”
–  defects didn’t make it to the test harness
–  overloaded system ignored data that was wrong


3-6


Automation +

DSTL
structured
testware
architecture

Dis

po
scr sabl
ipts e

execution
manual
comparison
testing

s
litie d
Uti ta loa
da
eg

loosen your
oracles

ost
& p ing
Pre ess
c
pro

traditional
test
automation

Me
tric
e.g s
EM .
TE

ETA,
monkeys

Pre and Post

1

2
5


3

4

3-7

6

Contents
Test status

3-8



Test status – pass or fail?
•  tool cannot judge pass or fail
–  only “match” or “no match”
–  assumption: expected results are correct

•  when a test fails (i.e. the software fails)
–  need to analyse the failure
•  true failure? write up bug report
•  test fault? fix the test (e.g. expected result)
•  known bug or failure affecting many automated tests?
–  this can eat a lot of time in automated testing
–  solution: additional test statuses
3-9

Test statuses for automation
Compare to
(true) expected outcome
expected fail outcome
don’t know / missing

No differences found Differences found
Pass
Fail
Expected Fail
Unknown
Unknown
Unknown

•  other possible additional test statuses
–  environment problem (e.g. network down, timeouts)
–  set-up problems (files missing)
–  things that affect the tests that aren’t related to the
assessment of matching expected results
–  test needs to be changed but not done yet
3-10


Pre and Post

1

2

3

4

5


6

Summary: key points
• 
• 

Pre- and post processing to automate setup
and clear-up tasks
Test status is more than just pass / fail

3-11



4-Scripting


1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice

Scripting Techniques
Ref. Chapter 3: Scripting Techniques
4-1

1

2

4

5


3
6

Scripting

Contents
4-2



4-Scripting

•  implement your
testware architecture
•  to reduce costs
–  make it easier to build
automated tests
•  avoid duplication

•  greater return on
investment
–  better testing support
–  greater portability
•  environments & hardware
platforms

•  enhance capabilities

–  avoid excessive
maintenance costs
•  greater reuse of functional,
modular scripting

–  achieve more testing for
same (or less) effort
•  testing beyond traditional
manual approaches

best achieved by keywords or DSTL
4-3

1

2

4

5


3
6

Scripting

Contents
4-4



4-Scripting

Capture/replay = linear script
test
scripts
record
/edit

software under test

(manual) test
procedures

Test
Tool

instructions
and test data
testers

automators
4-5

About capture/replay
•  fair amount of effort to produce scripts
–  similar for each test procedure
–  subsequent editing may also be necessary

•  dominated by maintenance costs
–  scripts are exact mould for software
–  one software change can mean many scripts
change
–  script is linear – sequence of every step

4-6



4-Scripting

Example captured script
captured individual
interactions
with software

duplicated actions
result in duplicated
instructions

FocusOn ‘Open File’
Type ‘countries’
LeftMouseClick ‘Open’
FocusOn ‘Scribble’
SelectOption ‘List/Add Item’
FocusOn ‘Add Item’
Type ‘Sweden’
LeftMouseClick ‘OK’
SelectOption ‘List/Add Item’
FocusOn ‘Add Item’
Type ‘USA’
SelectOption ‘List/Move Item’

duplicated actions
result in duplicated
instructions

4-7

Structured scripting
low-level
“how to”
instructions

create
Test
Tool

testers

script
library
automators

software under test

high-level
instructions
and test data
test
(manual) test
scripts
procedures

4-8



4-Scripting

About structured scripting
•  script library for re-used scripts
–  part of testware architecture implementation
•  shared scripts interface to software under test
•  all other scripts interface to shared scripts

•  reduced costs
–  maintenance
•  fewer scripts affected by software changes

–  build
•  individual control scripts are smaller and easier to read
(a ‘higher’ level language is used)
4-9

Structured scripting example
test script
LeftMouseDouble ‘Scribble’
Call OpenFile(‘countries’)
Call AddItem(‘Sweden’)
Call AddItem(‘USA’)
Call MoveItem(4,1)
Call AddItem(‘Norway’)
Call DeleteItem(2)
Call DeleteItem(7)
FocusOn ‘Delete Error’
Call CloseSaveAs(‘countries2’)
SelectOption ‘File/Exit’

shared scripts
OpenFile

DeleteItem

::::::::::::::::::::::

AddItem
::::::::::::::::::::::

::::::::::::::::::::::

MoveItem
::::::::::::::::::::::

CloseSaveAs
::::::::::::::::::::::

4-10



4-Scripting

Usable (re-usable) scripts
•  To re-use a script, need to know:
–  what does this script do?
–  what does it need?
–  what does it deliver?
–  what state when it starts?
–  what state when it finishes?

•  Information in a standard place for every script
–  can search for the answers to these questions

4-11

Data driven
test data
data
files

control
scripts

create

Test
Tool

create

testers

script
library
automators

low-level
“how to”
instructions

software under test

(manual) test
procedures

high-level
instructions

4-12



4-Scripting

About data driven
•  test data extracted from scripts
–  placed into separate data files

•  control script reads data from data file
–  one script implements several tests by reading different
data files (reduces script maintenance per test)

•  reduced build cost
–  faster and easier to automate similar test procedures
–  many test variations using different data

•  multiple control scripts required
–  one for each type of test (with varying data)
4-13

Data-driven example

OpenDataFile(TESTCASEn)
ReadDataFile(RECORD)

Data file: TestCase1
FILE

ADD

MOVE DELETE

For each record
ReadDataFile(RECORD)
Case (Column(RECORD))

countries
Sweden
USA
FILE

Europe
Norway

ADD
4,1
France
Germany

MOVE DELETE

2
7
1,3
2,2
5,3

FILE:
OpenFile(INPUTFILE)
ADD:
AddItem(ITEM)

1


Control script
For each TESTCASE

MOVE:
MoveItem(FROM, TO)
DELETE:
DeleteItem(ITEM)

…..

Next record
Next TESTCASE

4-14


4-Scripting

Data-driven: example data file
FILE

ADD

MOVE DELETE

countries
Sweden
USA

Headings indicate actions
- but may not be read by a
control script (they could be
just “comments”)

4,1
Norway
Position represents data for the
action (data in the wrong
column will cause problems)

2
7

Tests are not identical – first one has 3 countries, 2 moves,
next test had 2 countries, 3 moves.
The test data drives the test for things it can deal with –
the script responds to data in a fixed position in the data file.

4-15

Keywords (basic)
single control script:
“interpreter” / ITE
control
script

create
Test
Tool
test
definitions
testers


script
library
automators

software under test

high-level
instructions
and test data
(manual) test
procedures

low-level “how to”
instructions and
keywords 4-16


4-Scripting

About keywords
•  single control script (Interactive Test Environment)
–  improvements to this benefit all tests (ROI)
–  extracts high-level instructions from scripts

•  ‘test definition’

Unit test: calculate
one interest payment

–  independent of tool scripting language
–  a language tailored to testers’ requirements
System test:
summarise interest
for one customer

•  software design
•  application domain
•  business processes

Acceptance test:
end of day run, all
interest payments

•  more tests, fewer scripts

4-17

Comparison of data files
data-driven approach
FILE
ADD MOVE DELETE SAVE
Europe
France
Italy
1,3
2,2
1
5,2
Test2

which is easier to
read/understand?


keyword approach
ScribbleOpen Europe
AddToList France Italy
MoveItem 1 to 3
MoveItem 2 to 2
DeleteItem 1
MoveItem 5 to 2
SaveAs Test2

what happens when
the test becomes
large and complex?

this looks more like
a test
4-18


4-Scripting

1

2

4

5


3
6

Scripting

Contents
4-19

Merged test procedure/test/ definition
Advanced keyword-driven action words
single control script:
“interpreter” / ITE
control
script

create
Test
Tool
test
test
procedures
definitions
/definitions
/
language for testers


script
library

low-level “how to”
instructions and
keyword scripts

software under test

high-level
instructions
and test data
(manual) test
procedures

4-20


4-Scripting

Domain Specific Test Language (DSTL)
•  test procedures and test definitions similar
–  both describe sequences of test cases
•  giving test inputs and expected results

•  combine into one document
–  can include all test information
–  avoids extra ‘translation’ step
–  testers specify tests regardless manual/automated
–  automators implement required keywords

4-21

Keywords in the test definition language
•  multiple levels of keywords possible
–  high level for business functionality
–  low level for component testing

•  composite keywords
–  define keywords as sequence of other keywords
–  gives greater flexibility (testers can define composite
keywords) but risk of chaos

•  format
–  freeform, structured, or standard notation
•  (e.g. XML)
4-22



4-Scripting

Example use of keywords
Create a new account, order 2 items and check out
Firstname

Surname

Email address

Password

Edward

Brown

ebrown@gmail.com

apssowdr

Item

Num Items

Check Price for Items

Order Item

1579

3

15.30

Order Item

2598

Create Account

12.99

Checkout

4-23

Documenting keywords
Name
Purpose
Parameters
Preconditions
Postconditions
Error
conditions
Example

Name for this keyword
What this keyword does
Any inputs needed, outputs produced
What needs to be true before using it,
where valid
What will be true after it finishes
What errors it copes with, what is
returned
An example of the use of the keyword

Source: Martin Gijsen. See also Hans Buwalda book & articles


4-24


4-Scripting

Example keyword documentation - 1
Name

Create account

Purpose

Creates a new account

Parameters

*First name: 2 to 32 characters
*Last name: 2 to 32 characters
*Email address: also serves as account id
*Password: 4 - 32 characters

Pre-conditions

Account doesn't exist for this person

Post-conditions

Account created (including email confirmation)
order Screen displayed

Error conditions
Example

(see example)

*mandatory

4-25

Example keyword documentation - 2
Name
Purpose
Parameters

Pre-conditions

Post-conditions
Error conditions
Example

Order item
Orders items from the online supplier
*Item number: 1000 to 9999, in catalogue
Number of items wanted: 1 to Max-for-item. If blank,
assumes 1
Valid account logged in
Items in stock (at least one)
Prices available for all items in stock
Total for all items ordered is available (for checkout)
Number of available items decreased by number ordered
Item(s) not in stock
(see example)

*mandatory


4-26


4-Scripting

Implementing keywords
•  ways to implement keywords
–  scripting language (of a tool)
–  programming language (e.g. Java)
–  use what your developers are familiar with!

•  ways of creating an architecture to support a
DSTL
–  commercial, open source or home-grown
framework
–  spreadsheet or database for test descriptions
4-27

Tools / frameworks
•  commercial tools
–  ATRT, Axe, Certify, eCATT, FASTBoX, Liberation,
Ranorex, TestComplete, TestDrive, Tosca
Testsuite

•  open source
–  FitNesse, JET, Open2Test, Power Tools, Rasta,
Robot Framework, SAFS, STAF, TAF Core

•  I can email you my Tool List
–  test execution and framework tools
–  info@dorothygraham.co.uk
4-28



4-Scripting

Execution-tool-independent framework
some tests run manually

Test
Tool

Another
Test
Tool

test
procedures
/definitions
tool independent

1

2
5

4-29

tool dependent


3

4

sut

tool independent
scripting language framework

software under test
software under test

script
libraries

6

Scripting

Summary

•  Objectives of good scripting
–  to reduce costs and enhance capabilities

•  Many types of scripting
–  e.g. capture playback, structured, data-driven, keyword

•  Keyword/DSTL the most sophisticated
–  yields significant benefits for investment

•  Increased tester productivity
–  tailored front end, test language, tool independence
4-30



Keyword-Driven Exercise
Exercise (Part 1)
Using only the keywords described below, specify two or three test cases each
comprising a short sequence of keywords for an online grocery store.
Keywords
The following keywords are available.
Keyword

Arguments

Basket should contain

“n items” where n is the number of items in the basket

Checkout using credit card

Card type, Card number, Security code, Expiry date

Create a valid user

Username, Password, Address, Postcode

Last order status should be

One of: “Delivery scheduled”, “Delivery due”,
“Delivered”

Login

Username, Password

Logout
Put items in basket

List of items

Example
The following is an example of short test case.
Test Case
Existing user can
place an order

Action

Argument

Argument

Login

Joe Shopper

Passwurd

Put items in basket

Carrots

Cabbage

…

Potatoes

Basket should
contain

3 items

Checkout using
credit card

Mastercard

1234 5678 0123

…

123

01-11

Logout
Note that “…” in the Action column indicate a continuation line, i.e. row contains more
arguments for the previous keyword. Multiple continuation lines are allowed.
Exercise (Part 2)
Design 2 new additional keywords suitable for testing this online grocery store and
specify one or two additional test cases that use your new keywords.

© Grove Consultants

ATT090117

Page 1 of 3

Keyword-Driven Exercise

Test Case


Action

Argument

ATT090117

Argument

Page 2 of 3

Keyword-Driven Solution

Test Case
New user can
create account and
place order

Action

Argument

Argument

Create a valid user

Joe Shopper

Passwurd

…

6 Lower Road
Worcester

WR1 2AB

Put items in basket

Cornflakes

Milk

…

Tea bags

Coffee

…

Sugar

Biscuits

Basket should
contain

6 items

Checkout using
credit card

Mastercard

1234 5678 0123

…

123

01-11

Login

Joe Shopper

Passwurd

Last order status
should be

Delivery scheduled

Logout
Existing user can
query order

Logout


ATT090117

Page 3 of 3

5-Comparison


1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice

Automated Comparison
Ref. Chapter 4: Automated Comparison
5-1

1

2

4

5


3
6

Comparison

Contents
Test sensitivity

5-2



5-Comparison

Perverse persistence: Michael Williamson
–  testing Webmaster tools at Google (new to testing)
–  QA used Eggplant (image processing tool)
–  new UI broke existing automation
–  automate 4 or 5 functions
–  comparing bitmap images – inaccurate and slow
–  testers had to do automation maintenance
•  not worth developers learning tool’s language

–  after 6 months, went for more appropriate tools
–  QA didn’t use the automation, tested manually!
•  tool was just running in the background
5-3


Checking versus testing
–  checking confirms that things are as we think
•  e.g. check that the code still works as before

–  testing is a process of exploration, discovery,
investigation and learning
•  e.g. what are the threats to value to stakeholders, give
information

–  checks are machine-decidable
•  if it’s automated, it’s probably a check

–  tests require sapience
•  including “are the checks good enough”
Source: Michael Bolton, www.developsense.com/blog/2009/08/testing-vs-checking/
5-4



5-Comparison

General comparison guidelines
• 
• 
• 
• 
• 
• 

keep as simple as possible
well documented
standardise as much as possible
avoid bit-map comparison
poor comparisons destroy good tests
divide and conquer:
–  use a multi-pass strategy
–  compare different aspects in each pass
5-5

Two types of comparison
•  dynamic comparison
–  done during test
execution
–  performed by the test tool
–  can be used to direct the
progress of the test
•  e.g. if this fails, do that
instead

–  fail information written to
test log (usually)

•  post-execution
comparison
–  done after the test
execution has completed
–  good for comparing files
or databases
–  can be separated from
test execution
–  can have different levels
of comparison
•  e.g. compare in detail if all
high level comparisons
pass
5-6



5-Comparison

Comparison types compared
scribble1.scp

Test script:
- test input
- comparison
instructions

dynamic comparison
Error message
as expected?

log.txt

Log

countries.dcm

Initial
Document

Scribble

countries2.dcm

Edited
Document

countries2.dcm

Expected
Output

Compare
post-execution
comparison

diffs.txt

Differences
5-7

Comparison process
•  few tools for post-execution comparison
•  simple comparators come with operating
systems but do not have pattern matching
–  e.g, Unix ‘diff’, Windows ‘UltraCompare’

•  text manipulation tools widely available
–  sed, awk, grep, egrep, Perl, Tcl, Python

•  use pattern matching tools with a simple
comparator to make a ‘comparison process’
•  use masks and filters for efficiency
5-8



5-Comparison

1

2

4

5


3
6

Comparison

Contents
Test sensitivity

5-9

Test sensitivity
•  the more data there is available:
–  the easier it is to analyse faults and debug

•  the more data that is compared:
–  the more sensitive the test

•  the more sensitive a test:
–  the more likely it is to fail
–  (this can be both a good and bad thing)

5-10



5-Comparison

Sensitive versus specific(robust) test
Test is supposed
to change only
this field

Specific test
verifies this
field only

Test outcome

Unexpected
change occurs

Sensitive test
verifies the
entire outcome
5-11

Too much sensitivity = redundancy
Three tests,
each changes
a different field

If all tests are
specific, the
unexpected
change is
missed

Test
outcome
Unexpected
change occurs
for every test

If all tests are
sensitive, they
all show the
unexpected change
5-12



5-Comparison

Using test sensitivity
•  sensitive tests:
–  few, at high level
–  breadth / sanity checking tests
–  good for regression / maintenance

•  specific/robust tests:
–  many, at detailed level
–  focus on specific aspects
–  good for development

1

2

3

4

5

A good test
automation strategy
will plan
a combination of
sensitive and specific
tests
5-13


6

Comparison

Summary: key points
• 
• 
• 

Balance dynamic and post-execution
comparison
Balance sensitive and specific tests
Use masking and filters to adjust
sensitivity
5-14



Comparison Illustration

The following examples illustrate some of the problems in automated comparison.
Below are the expected outcome and the actual outcome for a test. Do they match? I.e.
does the test pass? The expected results were taken from a previous test that was
confirmed as correct.
Test objective: Ensure that the core processing is correct, i.e. the correct error messages
are produced, the progress of the order is correctly logged, and the right number of things
have been ordered. It doesn’t matter when the test is run, what specific things are ordered,
or the exact number assigned to the order. The total should be correct for the things that
have been ordered, however.
Test input (high level conditions): Log on as existing customer (software gets the first one
in the list of existing customers). Order one thing that is not in stock (from a pre-defined
table.) Order three things that are in stock (3 things at random locations in the stock
database). Total the cost of items ordered.
Expected outcome
Order number X43578
Date: 24 Jan 2008
Time 11:02
Message: “Log-on accepted, J. Smith”
Message: “Add to basket, 1 toaster”
Message: “Out of stock, mountain bike”
Message: “Add to basket, 1 computer mouse”
Message: “Add to basket, 1 pedometer”
Add to total: £15.95
Message: “Check out J Smith”
Message: “Total due is £35.89”
Actual outcome
Message: “Log-on accepted, M. Jones”
Order number X43604
Message: “Out of stock, anti-gravity boots”
Message: “Add to basket, 1 nose stud”
Message: “Add to basket, 1 car booster seat”
Date: 18 Feb 2008
Time 13:08
Has this test passed or failed? How long did it take you to decide? Would automated
comparison be better / quicker?
© Dorothy Graham

STA110126

Page 1 of 4

Computer matching attempt No. 1:
Expected outcome

Actual outcome

Test run number X43578


Comparison
result
FAIL

Date: 24 Jan 2008


FAIL

Time 11:02


FAIL



FAIL



FAIL



FAIL


Add to total: “£69.95”

FAIL



PASS



FAIL



FAIL



FAIL


Date: 18 Feb 2008

FAIL

Message: “Total due is £35.89

Time 13:08

FAIL

Well, this isn’t terribly helpful, is it! The only thing that passed was just a coincidence
(ordering the same item in the two tests).

What filters could we use to get the computer’s simple comparison to give a more
meaningful result?
We can see that some things would have a better chance of matching if the results were in
the same order. For example, Date and Time come last in the actual outcome but nearly
first in our expected outcome.
Let’s apply a filter to both sets of results to put the individual lines in alphabetical order.
Note that this now destroys the sequence of steps as a test (this may be OK for the
purposes of comparing the individual steps).

© Dorothy Graham

STA110126

Page 2 of 4

After applying the filter:
Sort into alphabetical order:
Expected outcome

Actual outcome



Comparison
result
FAIL



FAIL



FAIL

Date: 24 Jan 2008

Date: 18 Feb 2008

FAIL



FAIL



FAIL



FAIL



PASS



FAIL



FAIL



FAIL



FAIL

Time 11:02

Time 13:08

FAIL

We don’t seem to be much further forward now, although a different item has now
passed! There are two items that would have matched if they had been in a row higher or
lower.
There is also something very strange about this result – the thing that has passed is
‘Message: “Check out J Smith”’. But we logged in as two different people!
So the one thing shown as a Pass is actually a bug!!

What other filters could we apply to get a better result from automated comparison?

© Dorothy Graham

STA110126

Page 3 of 4

Here are some other filters we could apply:
Filter: After {Time} change if in the correct format to <nn:nn>
After {Test run number} change if in the correct format to <Xnnnnn>
After {Date} change if in the correct format to <nn Aaa nn>
After {Message: “Add to basket, } change to <item> all text before {”}
After {Message: “Out of stock, } change to <item> all text before {“}”
Here is our result after applying these filters:
Expected outcome

Actual outcome



Comparison
result
FAIL



FAIL



FAIL

Date: <nn Aaa nn>

Date: <nn Aaa nn>

Message: “Add to basket, <item>”




PASS
PASS
PASS





PASS
PASS



FAIL

Message: “Out of stock, <item>”

Message: “Out of stock, <item>”

PASS



FAIL

Test run number <Xnnnnn>

Test run number <Xnnnnn>

Time <nn:nn>

Time <nn:nn>

PASS
PASS

We were not interested in checking what items were ordered – if we aren’t interested in
the amounts either, we could apply further filters to the amounts?
Note that there are two aspects where simple comparisons such as those shown here are
not really adequate.
We want to know that the total for the amount ordered is correct (by the way, the other
bug is that the total for the Actual outcome should be £79.89). In order to check this
automatically, we would need to define a variable, add the individual item amounts to it,
and check that its total was the same as that calculated by the application. This can
certainly be done, but it takes additional effort to program it into the comparison process!
Similarly, we could store the name used at log-on and compare it to the name used at
check out, and this would find that bug (but also requires additional work).
Conclusion: automated comparison is not trivial!

© Dorothy Graham

STA110126

Page 4 of 4

6-Advice


1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice

Final Advice and Direction
6-1

What next?
•  we have looked at a number of ideas about
test automation today
•  what is your situation?
–  what are the most important things for you now?
–  where do you want to go?
–  how will you get there?

•  make a start on your test automation
strategy now
–  adapt it to your own situation tomorrow
6-2



6-Advice

Strategy exercise
•  your automation strategy / action plan
–  review your objectives for today (p1)
–  review your “take-aways” so far (p2)
–  identify the top 3 changes you want to make to your
automation (top of p3)
–  note your plans now on p3

•  discuss with your neighbour or small group
–  exchange emails, keep in touch
–  form a support group for each other
6-3

6-4



6-Advice

Dealing with high level management
•  management support
–  building good automation takes time and effort
–  set realistic expectations

•  benefits and ROI
–  make benefits visible (charts on the walls)
–  metrics for automation
•  to justify it, compare to manual test costs over iterations
•  on-going continuous improvement
–  build cost, maintenance cost, failure analysis cost
–  coverage of system tested
6-5

Dealing with developers
•  critical aspect for successful automation
–  automation is development
•  may need help from developers
•  automation needs development standards to work
–  testability is critical for automatability
–  why should they work to new standards if there is “nothing in it
for them”?

–  seek ways to cooperate and help each other
•  run tests for them
–  in different environments
–  rapid feedback from smoke tests

•  help them design better tests?
6-6



6-Advice

Standards and technical factors
•  standards for the testware architecture
–  where to put things
–  what to name things
–  how to do things
•  but allow exceptions if needed

•  new technology can be great
–  but only if the context is appropriate for it (e.g.
Model-Based Testing)

•  use automation “outside the box”
6-7

On-going automation
•  you are never finished
–  don’t “stand still” - schedule regular review and refactoring of the automation
–  change tools, hardware when needed
–  re-structure if your current approach is causing
problems

•  regular “pruning” of tests
–  don’t have “tenured” test suites
•  check for overlap, removed features
•  each test should earn its place
6-8



6-Advice

Information and web sites
–  www.AutomatedTestingInstitute.com
•  TestKIT Conference, Autumn, near Washington DC
•  Test Automation certificate (TABOK)

–  tool information
•  commercial and open source: http://testertools.com
•  open source tools
–  www.opensourcetesting.org
–  http://sourceforge.net
–  http://riceconsulting.com (search on “cheap and free tools”)

–  www.ISTQB.org
•  Expert level in Test Automation (in progress)
6-9

1

2

3

4

5


6

Summary: successful test automation
•  assigned responsibility for automation tasks
•  realistic, measured objectives (testing ≠ automation)
•  technical factors – testware architecture, levels of
abstraction, DSTL, scripting and comparison, pre and
post processing
•  management support, ROI, continuous improvement
6-10



6-Advice

any more questions?
please email me!
info@DorothyGraham.co.uk
Thank you for coming today
I hope this will be useful for you
All the best in your automation!
6-11



32

BETTER SOFTWARE

JULY/AUGUST 2009

www.StickyMinds.com

“Why automate?”

This seems such
an easy question to answer; yet many
people don’t achieve the success they
hoped for. If you are aiming in the wrong
direction, you will not hit your target!
This article explains why some
testing objectives don’t work for automation, even though they may be very
sensible goals for testing in general. We
take a look at what makes a good test
automation objective; then we examine
six commonly held—but misguided—
objectives for test execution automation,
explaining the good ideas behind them,
where they fail, and how these objectives
can be modified for successful test automation.

Good Objectives for Test
Automation
A good objective for test automation
should have a number of characteristics.
First of all, it should be measurable so
that you can tell whether or not you
have achieved it.
Objectives for test automation should
support testing activities but should not
be the same as the objectives for testing.
Testing and automation are different and
distinct activities.
Objectives should be realistic and
achievable; otherwise, you will set yourself up for failure. It is better to have
smaller-scale goals that can be met than
far-reaching goals that seem impossible.
Of course, many small steps can take
you a long way!
Automation objectives should be
both short and long term. The shortterm goals should focus on what can be
achieved in the next month or quarter.
The long-term goals focus on where you
want to be in a year or two.
Objectives should be regularly revised
in the light of experience.

Misguided Objectives for
Test Automation
OBJECTIVE 1: FIND MORE BUGS
Good ideas behind this objective:
tomated testing should find them
quicker.
run more tests and find even more
bugs.

so we should also find bugs in
the parts we weren’t able to test
manually.
Basing the success of automation on
finding bugs—especially the automation of regression tests—is not a good
thing to do for several reasons. First, it
is the quality of the tests that determines
whether or not bugs are found, and this
has very little, if anything, to do with
automation. Second, if tests are first run
manually, any bugs will be found then,
and they may be fixed by the time the
automated tests are run. Finally, it sets
an expectation that the main purpose of
test automation is to find bugs, but this
is not the case: A repeated test is much
less likely to find a new bug than a new
test. If the software is really good, automation may be seen as a waste of time
and resources.
Regression testing looks for unexpected, detrimental side effects in unchanged software. This typically involves running a lot of tests, many of
which will not find any defects. This is
ideal ground for test automation as it
can significantly reduce the burden of
this repetitive work, freeing the testers
to focus on running manual tests where
more defects are likely to be. It is the
testing that finds bugs—not the automation. It is the testers who may be able to
find more bugs, if the automation frees
them from mundane repetitive work.
The number of bugs found is a misleading measure for automation in any
case. A better measure would be the percentage of regression bugs found (compared to a currently known total). This
is known as the defect detection percentage (DDP). See the StickyNotes for
more information.
Sometimes this objective is phrased
in a slightly different way: “Improve
the quality of the software.” But identifying bugs does nothing to improve
software—it is the fixing of bugs that
improves the software, and this is a development task.
If finding more bugs is something that
you want to do, make it an objective for
measuring the value of testing, not for
measuring the value of automation.
Better automation objective: Help teswww.StickyMinds.com

ters find more regression bugs (so fewer
regression failures occur in operation).
This could be measured by increased
DDP for regression bugs, together with
a rating from the testers about how well
the automation has supported their objectives.

OBJECTIVE 2: RUN REGRESSION TESTS
OVERNIGHT AND ON WEEKENDS
nings and weekends).
“while we sleep.”
At first glance, this seems an excellent
objective for test execution automation,
and it does have some good points.
Once you have a good set of automated regression tests, it is a good idea
to run the tests unattended overnight
and on weekends, but resource use is not
the most important thing.
What about the value of the tests
that are being run? If the regression
tests that would be run “off peak” are
really valuable tests, giving confidence
that the main areas of the system are still
working correctly, then this is useful.
But the focus needs to be on supporting
good testing.
It is too easy to meet this stated objective by just running any test, whether it
is worth running or not. For example, if
you ran the same one test over and over
again every night and every weekend,
you would have achieved the goal as
stated, but it is a total waste of time
and electricity. In fact, we have heard of
someone who did just this! (We think he
left the company soon after.)
Of course, automated tests can be run
much more often, and you may want
some evidence of the increased test execution. One way to measure this is using
equivalent manual test effort (EMTE).
For all automated tests, estimate how
long it would have taken to run those
tests manually (even though you have no
intention of doing so). Then each time
the test is run automatically, add that
EMTE to your running total.
Better automation objective: Run the
most important or most useful tests, employing under-used computer resources
when possible. This could be partially
JULY/AUGUST 2009

BETTER SOFTWARE

33

measured by the increased use of resources and by EMTE, but should also
include a measure of the value of the
tests run, for example, the top 25 percent of the current priority list of most
important tests (priority determined by
the testers for each test cycle).

OBJECTIVE 3: REDUCE TESTING STAFF
tool, so we should be able to save
elsewhere.
and staff costs are high.
This is an objective that seems to
be quite popular with managers. Some
managers may go even further and think
that the tool will do the testing for them,
so they don’t need the testers—this is
just wrong. Perhaps managers also think
that a tool won’t be as argumentative as
a tester!
It is rare that staffing levels are reduced when test automation is introduced; on the contrary, more staff are
usually needed, since we now need
people with test script development skills
in addition to people with testing skills.
You wouldn’t want to let four testers go
and then find that you need eight test automators to maintain their tests!
Automation supports testing activities; it does not usurp them. Tools cannot
make intelligent decisions about which
tests to run, when, and how often. This
is a task for humans able to assess the
current situation and make the best use
of the available time and resources.
Furthermore, automated testing is
not automatic testing. There is much
work for people to do in building the automated tests, analyzing the results, and
maintaining the testware.
Having tests automated does—or at
least should—make life better for testers.
The most tedious and boring tasks are
the ones that are most amenable for automation, since the computer will happily do repetitive tasks more consistently
and without complaining. Automation
can make test execution more efficient,
but it is the testers who make the tests
themselves effective. We have yet to see
a tool that can think up tests as well as a
human being can!
34

BETTER SOFTWARE

JULY/AUGUST 2009

The objective as stated is a management objective, not an appropriate objective for automation. A better management objective is “Ensure that everyone
is performing tasks they are good at.”
This is not an automation objective
either, nor is “Reducing the cost of
testing.” These could be valid objectives,
but they are related to management, not
automation.
Better automation objective: The total
cost of the automation effort should be
significantly less than the total testing effort saved by the automation. This could
be partially measured by an increase in
tests run or coverage achieved per hour
of human effort.

OBJECTIVE 4: REDUCE ELAPSED TIME
FOR TESTING
way we can save time is good.
testing will help overall.

This one seems very sensible at first
and sometimes it is even quantified—
“Reduce elapsed time by X%”—which
sounds even more impressive. However,
this objective can be dangerous because
of confusion between “testing” and “test
execution.”
The first problem with this objective is that there are much easier ways
to achieve it: run fewer tests, omit long
tests, or cut regression testing. These are
not good ideas, but they would achieve
the objective as stated.
The second problem with this objective is its generality. Reducing the
elapsed time for “testing” gives the impression we are talking about reducing
the elapsed time for testing as a whole.
However, test execution automation
tools are focused on the execution of
the tests (the clue is in the name!) not
the whole of testing. The total elapsed
time for testing may be reduced only if
the test execution time is reduced sufficiently to make an impact on the whole.
What typically happens, though, is that
the tests are run more frequently or
more tests are run. This can result in
more bugs being found (a good thing),
that take time to fix (a fact of life), and
www.StickyMinds.com

increase the need to run the tests again
(an unavoidable consequence).
The third problem is that there are
many factors other than execution that
contribute to the overall elapsed time
for testing: How long does it take to set
up the automated run and clear up after
it? How long does it take to recognize a
test failure and find out what is actually
wrong (test fault, software fault, environment problem)? When you are testing
manually, you know the context—you
know what you have done just before
the bug occurs and what you were doing
in the previous ten minutes. When a tool
identifies a bug, it just tells you about the
actual discrepancy at that time. Whoever
analyzes the bug has to put together the
context for the bug before he or she can
really identify the bug.
In figures 1 and 2, the blocks represent the relative effort for the different
activities involved in testing. In manual
testing, there is time taken for editing
tests, maintenance, set up of tests, executing the tests (the largest component
of manual testing), analyzing failures,
and clearing up after tests have completed. In figure 1, when those same tests
are automated, we see the illusion that
automating test execution will save us
a lot of time, since the relative time for
execution is dramatically reduced. However, figure 2 shows us the true picture—
total elapsed time for testing may actually increase, even though the time for
test execution has been reduced. When
test automation is more mature, then the
total elapsed time for all of the testing
activities may decrease below what it
was initially for manual testing. Note
that this is not to scale; the effects may
be greater than we have illustrated.
We now can see that the total elapsed
time for testing depends on too many
things that are outside the control or influence of the test automator.
The main thing that causes increased
testing time is the quality of the software—the number of bugs that are already there. The more bugs there are,
the more often a test fails, the more bug
reports need to be written up, and the
more retesting and regression testing
are needed. This has nothing to do with
whether or not the tests are automated or
manual, and the quality of the software

is the responsibility of the developers,
not the testers or the test automators.
Finally, how much time is spent maintaining the automated tests? Depending
on the test infrastructure, architecture,
or framework, this could add considerably to the elapsed time for testing.
Maintenance of the automated tests for
later versions of the software can consume a lot of effort that also will detract
from the savings made in test execution.
This is particularly problematic when
the automation is poorly implemented,
without thought for maintenance issues
when designing the testware architecture. We may achieve our goal with the
first release of software, but later versions may fail to repeat the success and

may even become worse.
Here is how the automator and tester
should work together: The tester may
request automated support for things
that are difficult or time consuming, for
example, a comparison or ensuring that
files are in the right place before a test
runs. The automator would then provide utilities or ways to do them. But the
automator, by observing what the tester
is doing, may suggest other things that
could be supported and “sell” additional
tool support to the tester. The rationale
is to make life easier for the tester and
to make the testing faster, thus reducing
elapsed time.
Better automation objective: Reduce
the elapsed time for all tool-supported

Figure 1

Figure 2

www.StickyMinds.com

testing activities. This is an ongoing
objective for automation, seeking to
improve both manual and existing automated testing. It could be measured by
elapsed time for specified testing activities, such as maintenance time or failure
analysis time.

OBJECTIVE 5: RUN MORE TESTS
better coverage.
must be better.
More is not better! Good testing is
not found in the number of tests run, but
in the value of the tests that are run. In
fact, the fewer tests for the same value,
the better. It is definitely the quality of
the tests that counts, not the quantity.
Automating a lot of poor tests gives you
maintenance overhead with little return.
Automating the best tests (however many
that is) gives you value for the time and
money spent in automating them.
If we do want to run more tests, we
need to be careful when choosing which
additional tests to run. It may be easier
to automate tests for one area of the
software than for another. However, if it
is more valuable to have automated tests
for this second area than the first, then
automating a few of the more difficult
tests is better than automating many of
the easier (and less useful) tests.
A raw count of the number of automated tests is a fairly useless way of
gauging the contribution of automation
to testing. For example, suppose testers
decide there is a particular set of tests
that they would like to automate. The
real value of automation is not that the
tests are automated but the number of
times they are run. It is possible that the
testers make the wrong choice and end
up with a set of automated tests that
they hardly ever use. This is not the fault
of the automation, but of the testers’
choice of which tests to automate.
It is important that automation is
responsive, flexible, and able to automate different tests quickly as needed.
Although we try to plan which tests to
automate and when, we should always
start automating the most important
tests first. Once we are running the tests,
JULY/AUGUST 2009

BETTER SOFTWARE

35

Managing Successful Test Automation

Managing Successful Test Automation

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (15)

Semelhante a Managing Successful Test Automation

Semelhante a Managing Successful Test Automation (20)

Mais de TechWell

Mais de TechWell (20)

Último

Último (20)

Managing Successful Test Automation