Randy Rice of http://www.riceconsulting.com presents 14 questions you can ask to know if a software application or function is ripe for automation. Also presented are metrics for test automation ROI.

1. THE LOW-HANGING FRUIT
OF TEST AUTOMATION -
WHAT DO WE AUTOMATE
FIRST?
RANDALL RICE, CTAL
WWW.RICECONSULTING.COM
March 19, 2014

2. 2
BIO - RANDALL W. RICE
• Over 35 years experience in building and testing
information systems in a variety of industries and
technical environments
• ASTQB Certified Tester – Foundation level, Advanced
level (Full)
• Director, American Software Testing Qualification
Board (ASTQB)
• Chairperson, 1995 - 2000 QAI’s annual software
testing conference
• Co-author with William E. Perry, Surviving the Top
Ten Challenges of Software Testing and Testing Dirty
Systems
• Principal Consultant and Trainer, Rice Consulting
Services, Inc.
• www.riceconsulting.com & randallrice.blogspot.com

3. 3
FEASIBILITY
1. Is it easy (or even feasible) to automate the function/
component?
• Problematic issues
• Custom objects
• Non-standard objects
• Objects with dynamic attributes (session IDs, etc.)
• Streaming, dynamic data (think stock tickers and other data
that is changing even while you are testing it)
• The tool doesn’t handle a certain type of object or component.

4. 4
CONTROL
2. Who controls/develops the application?
• Are any components (or even the entire application) vendor-
developed?
• How stable is the code?
• Are there a lot of hotfixes or maintenance releases
• If so, do you know when they occur?
• Do you have prior knowledge of changes?
• Some vendors have problems in creating and distributing
release notes, even to their own testers.

5. 5
OTHER QUESTIONS
3. Is the function easily repeatable?
• Do you feel like a robot testing it?
4. Is the function easily performed?
• Does it take a lot of set-up?
• Are there a lot of steps to perform?
• Long scripts are hard to maintain?
5. How frequently is it used?
• X times per day (or hour)
6. How long does the function take to test manually, with all
test conditions?

6. 6
NUMBER OF TESTS
Timeperfunctionalexecution
Number of times executed per test
Long term ROI (not so good)

7. 7
TEST FREQUENCY
Timepertestsuiteexecution
Number of times tested per day, week, etc.
Good choice
Long term ROI (not so good)
Much depends on the length of the test.
5 test conditions vs. 500 obviously makes a big
Difference!

8. 8
REAL-LIFE EXAMPLE
• A financial services client had a simple
function that took about 5 minutes per
condition to test.
• However, it took the tester all day to test it
due to the number of conditions.
• We spent 3 days to automate it, with the
result being 15 minutes to test all
conditions!

9. 9
HELPFUL TEST
AUTOMATION METRICS (2)
• Effort saved by test automation (manual vs. automation)
• Effort needed to automate new tests
• Effort needed to analyze failed automated tests
• Effort to maintain automated tests
• Test coverage achieved by automated tests

10. 10
EQUIVALENT MANUAL
TEST EFFORT
• Also known as EMTE
• How much time would it take if the automated tests were
performed manually?

11. 11
EMTE EXAMPLE
• Automated test takes 1 hour to run and are performed for
10 cycles.
• Manual version of test takes 4 days to run.
• EMTE = 40 days
• 4 days X 10 cycles

12. 12
RETURN ON
INVESTMENT (ROI)
Time
Effort
Spent on
Automating
Tests
Development
Effort on
Production
Code
Initial
Effort
Increased
Effort (Hump)
Reduced
Effort
Source: xUnit Patterns by Gerald Meszaros
Saved Effort

13. 13
RETURN ON INVESTMENT (2)
- NOT SO GOOD
Time
Effort
Spent on
Automating
Tests
Development
Effort on
Production
Code
Initial
Effort
Increased
Effort (Hump)
Ongoing
Effort
Source: xUnit Patterns by Gerald Meszaros
Saved Effort

14. 14
MORE QUESTIONS
7. Does the function require:
• Human judgment for outcomes?
• Creative types of testing
• “Off the scripted path?”
8. Are you looking for new defects?
• Test automation is more confirmatory than discovery in
nature.
• i.e., the defects found by automation are normally regression-
type defects.

15. 15
NEW OR MAINTENANCE?
9. Is this new software or maintenance?
• Great debate: When should the automation be created –
during the project or after?
• Since functional changes ripple through automation, great
care must be taken not to automate too early.
• However, automation can be helpful during the final phases of
a project to speed up regression testing.
• Keep in mind it takes time to create test automation and the
distraction can actually turn into a project risk!

16. 16
WHO AND WHEN?
10. Who will create and use the automation?
• Simple capture/playback (when it works) can be created by
testers.
• When problems are encountered, technical help at the
developer level is often needed.
11. Which phase of testing?
• Maintenance testing is ideal
• UAT is not good (you want humans testing)
• System testing is a possible win
• Unit test automation is ideal and essential

17. 17
NEED FOR PRECISION
12. What is the need for precision?
• Exact regression testing requires
automation
• One mistake can invalidate the entire
test
• Otherwise, you are doing “pseudo-
regression” testing
• What is the risk?
• If exact testing is needed, such as
medical devices, avionics, etc.,
automation can help mitigate risks
• WARNING: test automation is “software
testing software”, therefore the test
scripts may have bugs

18. 18
KNOWLEDGE AND
STABILITY
13. Do you have a sufficient way to compare the test results
for pass/fail?
• You must have a source (non-human)
14. How stable is the function to be tested?
• This is an irony
• Functional errors (which you want to find) will cause the
scripts to fail
• However, they will make test automation difficult because you
will need a way to deal with errors without stopping other tests
in the suite.

19. 19

20. 20
CONTACT INFORMATION
Randall W. Rice, CTAL
Rice Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73170
Ph: 405-691-8075
Fax: 405-691-1441
Web site: www.riceconsulting.com
e-mail: rrice@riceconsulting.com