Provides overview of Software Functional Test Automation, What tool you should use? What are the benefits? How to select tool that best fit you?
Compiled after going through 50 plus slides from internet
2. Contents
Overview of STA
BEP and ROI
Tools
Automation Framework
3. Contents
Overview of STA Choosing the right tool – The Strategic
History Steps
Myths and Truths Tools Classification
Why Automation Projects Fail? Short List Tools – Functional
What is TA? Short List Tools – Performance
Why TA is needed? Evaluate Vendors
Manual vs. Automated Functional Test Tools - Analysis
Pros & Cons Feasibility Analysis
What to Automate? Automation Framework
What Not to Automate? Where to start?
BEP and ROI What is AF?
BEP Benefits of AF
How does it work?
BEP - Example
Architecture
ROI
Framework Approaches
Classic ROI
Record and Playback
Real ROI Script Based Approach
Benefits of ROI Keyword Driven Approach
ROI Calculator Data Driven Approach
Hybrid Keyword and Data Driven Approach
Tools
Choosing the right tool – The Strategic
Approach
4. Contents
Overview
History
Myths and Truths
Why Automation Projects Fail?
What is TA?
Why TA is needed?
Manual vs. Automated
Pros & Cons
What to Automate?
What Not to Automate?
5. History
Moving swiftly past the hype
Historically Automation is perceived as a “Silver Bullet” of the Testing
world
“The term has been adopted into a general metaphor, where “silver bullet”
refers to any straight forward solution perceived to have extreme
effectiveness. The phrase typically appears with an expectation that some
new technology or practice will easily cure a major prevailing problem”.
7. Myths and Truths
1. Test Automation is simple, that every tester can do
it
Promoted by the Sales people by simply saying:
Record the script
Enhance the script by adding functions and data driving
Run the scripts
Report results
Under this influence QA manager
can proudly say “All our testers
are doing test automation”.
8. Myths and Truths
But in Reality: TA is a software development task
It should be designed, developed and tested
Need to have some kind of a programming
background to implement test automation. TA is not
as
complex as C++/C#/Java development.
TA components are assets that should be
treated like application source code
Don’t fall into tool vendor sales pitch
…remember Record & Playback is not real test
automation
9. Myths and Truths
2. Commercial TA tools are expensive
Under the influence of this myth some
companies, especially the small ones:
Try to develop their own test automation tools
Use scripting languages like Perl and Ruby
Use shareware test tools
Do not consider test automation at all
10. Myths and Truths
But in Reality: Commercial TA tools are not that
expensive
Per seat license for the most expensive automation tool is
$8K, which can be used for 5 years.
Maintenance/Support fees are 20% of tool cost or $1,600 per
year
The cost of this tool is $8K/5+$1,600 = $3,200 per year
The automation developer cost with overhead is $100K per
year
The cost of this tool is just 3% of the person who uses it, but
productivity gain can be very significant
11. Myths and Truths
Learning from past experience
Truth: 92% FAIL to meet target ROI
Automation Projects
8%
40% Working
ROI
Failure
60% 32%
2004 2010 (Estimated)
Industry: Test Automation $1 Billion $ 6.3 Billion
(Net Worth)
Automation Projects $ 0.6 Billion $ 3.8 Billion
(Failure Cost)
Source 1: http://www.nytimes.com/2006/07/26/technology/26hewlett.html
Source 2: http://www.slideshare.net/Jonathon_Wright/hybrid-keyword-data-driven-automation-frameworks-jonathon-wright
12. Why do Automation Projects typically Fail?
IDT study(www.idtus.com)
Misc
Lack of Time
Tool 15%
37%
Incompatibility
11%
Lack Of
Budget
17% Lack of
Expertise
20%
13. Why do Automation Projects typically Fail?
Lack of defined automation
methodology
Automation is not treated as a
legitimate project with the necessary
planning / resources
Test Automation is typically performed
at the end of the SDLC
After the initial success the
automation scripts are not maintained
for future builds
14. Why do Automation Projects typically Fail?
Testers are typically untrained in test tools and
programming techniques
No modularization (reusable functions) in automation
scripts
Automated tests cases are usually designed based
on front end functionality (black box testing)
15. What is Software Test Automation?
Test Automation is the use of software to execute
tests without Human intervention
It refers to the activities and efforts that intend to
automate engineering tasks and operations in a
software test process using well-defined
strategies and systematic solutions.
Not like Rube Goldberg
cartoons
16. Why TA is needed?
Objectives:
To free engineers from tedious and redundant manual
testing operations
To speed up a software testing process, and to reduce
software testing cost and time during a software life cycle
To increase the quality and effectiveness of a software test
process by achieving pre-defined adequate test criteria in a
limited schedule
17. Key to Success
To reduce manual testing activities
and redundant test operations using
a systematic solution to achieve a
better testing coverage.
18. Manual vs. Automated Testing
Manual Testing:
Testing time is consuming and tedious
Inefficient in today’s shorter SDLC
Delay the ability to thoroughly test an application
Critical bugs escape undetected
What happens when multiple platforms involved
Automated Testing:
Higher efficiency that accelerate the testing cycle and promote
software quality
Optimizes software quality and testing efficiency by delivering
Reusability
Predictability and Consistency
Productivity
Enables accurate assessment of quality level
19. Pros and Cons
Pros:
Speed
Reusability
Accuracy
Run Anytime
Efficiency
20. Pros and Cons
Cons:
SignificantInvestment
Maintenance
Not as Robust
Error Detection
Cannot Think
21. What to Automate?
Regression Tests: Stabilized tests that verify stabilized functionality
Tests rerun often: Tests that are executed regularly vs. rarely
Tests that will not expire shortly: Most tests have a finite lifetime
during which its automated script must recoup the additional cost
required for its automation
Tedious/Boring tests:
tests with many calculations and number verifications
repetitive tests performing the same operations over and over
tests requiring many performance measurements
Just plain boring tests
Reliably repeatable
22. What NOT to Automate?
Unstable functionality: Not reliably repeatable
Rarely executed tests: poor Return-On-Investment
Tests that will soon expire: poor Return-On-Investment
Requiring in-depth business analysis:
some tests require so much business specific knowledge that it becomes
prohibitive time wise to include every verification required to make its
automated script robust enough to be effective
exceedingly complex tests are sometimes not possible to automate because
computers cannot think
23.
24. Contents
BEP and ROI
BEP
BEP - Example
ROI
PayBack Period
Classic ROI
Real ROI
Benefits of ROI
ROI Calculator
25. BEP
Break-Even Point (BEP) is the
point at which cost or expenses
and revenue are equal: there is no
net loss or gain
27. ROI
Return on Investment
ROI = BENEFIT/COST
ROI = (total benefit – total cost) / (total cost)
ROI = (cost of manual – cost of automation) / cost of
automation
Where,
Automation Cost = Price Of HW + Price of SW + Development
Cost + Maintenance Cost + Execution Cost
Manual Testing Cost = Development Cost + Maintenance Cost
+ Execution Cost
Looks right, Doesn’t it?
28.
29. Classic ROI
Problems with classic ROI calculation:
You can’t compare Automated Testing and Manual
Testing. They are not the same and they provide different
information about the AUT.
You can’t compare cost of multiple execution of
automated tests vs. manual tests. You would never dream
of executing that many test cases manually
So then…what is real ROI?
30. Real ROI
ROI value is not the value of Automation vs. Cost of
executing these tests manually
Automation ROI value is the benefit of this type of
testing, and it can be:
Reducing Time to Market
Increased Test Efficiency (Productivity)
Increased Test Effectiveness
31. Benefits of ROI
Reduced Time to Market
Products delivered quickly
Makes people available to work on other projects
Higher margins, if no competitive products in market
Productivity and Effectiveness
More testing gets done faster, increasing the odds of
finding defects
Defects found early have better chances of being
fixed
Manual Testers can concentrate on clever ways to
finding defects, instead of typing test inputs and
verify output.
32. Benefits of ROI
About 7% of bug fixes create new bugs, sometimes in
already tested parts of the system. With automation
you can rerun tests for those modules. This almost
never happens when testing is done manually.
33. ROI Calculator
ROI Calculator:
Source 1:
http://www.aspiresys.com/testautomationroi/index.php
Source 2:
http://www.elbrus.com/services/test_automation_roi_
calc/
36. Contents
Tools
Choosing the right tool – The Strategic Approach
Choosing the right tool – The Strategic Steps
Tools Classification
Short List Tools – Functional
Short List Tools – Performance
Evaluate Vendors
Functional Test Tools - Analysis
Feasibility Analysis
37. Tools
There is no single best testing tool;
rather, different tools are more or less
appropriate in different environments.
38. Tools
Over 300 Test Tools are available
(http://www.softwareqatest.com)
Load/Performance tools – 54
Web Functional/Regression – 60
Java Test tools - 48
Other Web tools – 76
Which tool is right for you?
39. Choosing the right tool – The Strategic
Approach
• Is there an organizational methodology for test
automation?
• Which applications/processes?
• What is the impact to current project schedules?
• What is the effort in maintaining automated
tests?
• What are the costs?
• What about tools integration?
• What about Continuous Integration?
40. Choosing the right tool – The Strategic Steps
• Step 1: Define and Refine Requirements
• Step 2: Communicate the Impact
• Step 3: Develop Evaluation Methodology
• Step 4: Select, Procure, and Implement
41. Step 1: Define and refine requirement
Create a list of organizational requirements
What problems do you want the tool to solve?
What capabilities will the tool need to be effective in
your environment? Other lifecycle tools?
What constraints, budgetary or otherwise?
Identify compatibility issues
What operating systems does your application
support?
What is the development environment?
Does your application integrate with third-party
software?
Does the application use custom controls?
42. Step 1: Define and refine requirement
Identify tool audience
Who will use the tool on a day-to-day basis?
What is the level and mix of user skill levels?
Is your organization willing to invest in training?
Define technical or business requirements
Does your organization have additional requirements?
Software standards
Technical standards
Procurement rules
Preferred vendor rules
43. Step 1: Define and refine requirement
Identify budget constraints
How much can we afford?
How much is this worth?
New requirements may surface based on research
Did not know about
Forgot to include
44. Step 2: Communicate the Impact
Automated testing is part of the larger strategic
application development endeavor
Communicate the effects of implementing a tool
Chance to discuss and mitigate concerns
How tool may change job description
Commitment to training
Implementation strategy
Discussion may imply additional requirements
Business, functional, technical, or operational
45. Step 3: Develop the evaluation methodology
How will tools be compared?
Are there specific features that may differentiate
one tool from another?
Are there specific things that can eliminate a tool
from consideration?
Preferred vendor list can reduce evaluation scope
Demos and evaluations are time-consuming
Identify a representative set of activities to accomplish
with the tool during the evaluation
46. Step 4: Select, procure, and implement
Make an informed selection
Follow organization’s procurement process
Develop the implementation plan
What
When
Why
Who
How
47. Step 4: Select, procure, and implement
Develop an implementation plan
Enterprise applications requiring multiple releases
Applications that must produce a consistent set of results
using stable data
These characteristics fully leverage reusability and predictability
benefits of automated testing
Take implementation one step at a time
Take time for training
Keep focus on staff issues and reactions
48. Step 4: Select, procure, and implement
Develop a test plan
Describes scope, approach, resources and schedule for
all automated and manual activities
Rule of Thumb: (Test scripts) 40% manual - 60% automated
Create and deploy your automated tests
Be selective with the automation of test scripts
Verify the most critical functionality
Are the most likely to expose defects
Are expensive or impossible to perform manually
Use the first automated suites you build for
Smoke testing
Regression testing
49. Decision?
How to calculate the cost of functional test automation
Labor costs of Labor costs of
Cost of test automation Cost of tool(s) script creation script maintenance
If a test script will be run every week for the next 2 years, automate the test if the
cost of automation is less than the cost of manually executing the test 104 times.
Automate if
Cost of manually executing the test as many
Cost of automation
times as the automated test will be executed
50. Tools Classification
Test Tool Types Basic Descriptions of Different Types of Test Tools
Test Information Systematic solutions and tools support test engineers and quality assurance people to
Management create, update, and maintain diverse test information, including test cases, test scripts,
test data, test results, and discovered problems.
Test Execution and Systematic solutions and tools help engineer set up and run tests, and collect and
Control validate test results.
Test Generation Systematic solutions and tools generate program tests in an automatic way.
Test Coverage Systematic solutions and tools analyze the test coverage during a test process based
Analysis on selected test criteria.
Performance Testing Systematic solutions and tools support program performance testing and performance
and Measurement measurement.
Software Simulators Programs are developed to simulate the functions and behaviors of external systems,
or dependent subsystems/components for a under test program.
Regression Testing Test tools support the automation performance of regression testing and activities,
including test recording and re-playing.
51. Tools Classification
Types of Test Tools Test Tool Vendors Test Tools
Problem Management Tools Rational Inc. ClearQuest, ClearDDTS
Microsoft Corp. PVCS Tracker
Imbus AG Imbus Fehlerdatenbank
Test Information Management Rautional Inc. TestManager
Tools
Mercury Interactive TestDirectory
Test Suite Management Tools Evalid TestSuiter
Rational Inc. TestFactory
SUN JavaTest, JavaHarness
White-Box Test Tools McCabe & Associates McCabe IQ2
Junit
IBM IBM COBOL Unit Tester
IBM ATC
- Coverage Assistant
- Source Audit Assistant
- Distillation Assistant
- Unit Test Assistant
52. Tools Classification
Test Execution Tools OC Systems Aprob
Softbridge ATF/TestWright
AutoTester AutoTester
Rational Inc. Visual Test, Rational Functional
Tester
SQA Robot
Mercury Interactive WinRunner, Quick Test Prof
Sterling Software Vision TestPro
Compuware QARun
Seque Software SilkTest
RSW Software Inc. e-Test
Cyrano Gmbh Cyrano Robot
Code Coverage Analysis Tools Case Consult Corp. Analyzer, Analyzer Java
OC Systems Aprob
IPL Software Product Group Cantata/Cantata++
ATTOL Testware SA Coverage
Compuware NuMega TruCoverage
Software Research TestWorks Coverage
Rational Inc PureCoverage
SUN JavaScope
ParaSoft TCA
Software Automation Inc Panorama
53. Tools Classification
Load Test and Performance Tools Rational Inc. Rational Performance Tester
InterNetwork AG sma@rtTest
Compuware QA-Load
Mercury Interactive LoadRunner
RSW Software Inc. e- Load
SUN JavaLoad
Seque Software SilkPerformer
Client/Server Solutions, Inc. Benchmark Factory
Regression Testing Tools IBM Regression Testing Tool(ARTT)
Distillation Assistant
GUI Record/Replay Software Research eValid
Mercury Interactive Xrunner
Astra Astra QuickTest
AutoTester AutoTester, AutoTester One
54. Short List Tools - Functional
Vendor Tool Test Suite - Companion Tools
Compuware TestPartner QACenter Enterprise Edition+
Empirix e-Tester e-TEST suite
Rational Functional
IBM Rational Suite
Tester
Mercury QuickTest Professional Quality Center
RadView WebFT TestView Suite
Seapine QA Wizard Pro TestTrack Pro
Borland
SilkTest SilkCentral Test Manager
(Segue)
55. Short List Tools - Performance
Vendor Tool Test Suite - Companion Tools
Compuware QALoad QACenter Enterprise Edition+
Empirix e-Load e-TEST suite
Rational Performance Rational Suite
IBM
Tester
Mercury LoadRunner Quality Center
RadView WebLOAD TestView Suite
Facilita Forecast ForecastWeb, ForecastNet, ForecastDB
Borland
SilkPerformer SilkCentral Test Manager
(Segue)
57. Functional Test Tools - Analysis
Tool Pros Cons
IBM/Rational •Built as Eclipse Plug-In with full IDE •Insufficient browser
Functional Tester and Java support support
(RFT) •Supports Web 2.0, Java or .NET •Licensed product
applications
•Full GUI Object Map repository
HP/Mercury Quick •Supports Web 2.0, Java or .NET •VisualBasic scripting is
Test Pro (QTP) applications limited
•Full GUI Object Map repository •No IDE (may change in
•Seamless integration with new release)
QualityCenter •Licensed Product
Selenium RC & •Good browser support •No GUI Object repository
IDE •Good language support (Java, •Only web application
Ruby,C# ) support
•Can be easily extended as JUnit
suite
•Open-source
58. Feasibility Analysis
FA Matrix Available
Operational Feasibility
Technical Feasibility
Economic Feasibility
Schedule Feasibility
61. Contents
Automation Framework
Where to start?
What is AF?
Benefits of AF
How does it work?
Architecture
Framework Approaches
Record and Playback
Script Based Approach
Keyword Driven Approach
Data Driven Approach
Hybrid Keyword and Data Driven Approach
62. Where to Start?
“Start Quick wins
SMALL should be NEVER expect
think BIG” avoided to
automate 100%
First find out Then you can work out
What needs to be tested? What needs be automated?
What can be tested? What can be automated?
What could be tested? What could be automated?
Focus on key
critical
“Under Keep it business
promise, simple, processes
Over deliver?” wherever
possible
63. What is Automation Framework?
Framework – independent of application or
environment under test
A Test Automation Framework is a set of
assumptions, concepts and tools that provide
support for Automated Software Testing.
A reusable set of libraries or classes for a software
system (or subsystem).
A correctly implemented Test Automation Framework
can further improve ROI by reducing the
development and maintenance costs.
64. Benefits of Framework
Ease of Use – easy to learn and easy to use
Time – faster than capture/replay and scripting approach
Maintainability – significantly reduces the test maintenance
effort
Reusability – due to modularity of test cases and library
functions
Manageability - effective test design, execution, and
traceability
Accessibility – to design, develop & modify tests whilst
executing
Availability – scheduled execution can run unattended on a
24/7 basis
Reliability – due to advanced error handling and scenario
recovery
Flexibility – framework independent of AUT or environment
Measurability – customizable reporting of test results ensure
65. How does it work?
Different Implementations
One Example of Keyword Driven Framework could be:
Spreadsheets, Spreadsheets, Spreadsheets
Test Objects
Keywords and Methods = Toast!
Parameters
Description or Call the 911?
67. Framework Approaches
Record and Playback
Script Based Approach
Keyword Driven Approach
Data Driven Approach
Hybrid Keyword and Data Driven Approach
68. Manual Testing – Looking back
+ easy & cheap to start
+ flexible testing
- expensive every execution
- no auto regression testing
- less coverage measurement
69. Record and Playback
+ flexible testing
- expensive first execution
+ auto regression testing
- fragile tests break easily
- less coverage measurement
70. Script Based Approach
+/- test impl. = programming
+ automatic execution
+ auto regression testing
- fragile tests break easily?
(depends on abstraction)
- less coverage measurement
71. Data Driven Approach
Automation is data-centric
User defines just data sets to drive tests with
Will have an external data source (DB tables, Excel
spreadsheets, XML for data sets)
Flow control (navigation) is normally done by the test
script not by the data source
Ex: data set exercises creation of new sales accounts
functionality; stored in a DB table account_data
CompanyName PrimarySalesPerson Street Zip City State
Genesis Inc. Phil Collins 5775 Main st 30075 Atlanta GA
RollingStones Inc. Mick Jagger Jr. 2332 Washington st 02111 Boston MA
73. Keyword/Action Driven Approach
Automation is action-centric
De-compose your test cases/modules into granular re-usable
keywords
The idea is for non-coders to be able to create automated test
cases with action keywords
User defines flow control of the test via action keywords
Example: Test Case “Verify Checking Account Balance”
1. Enter Username and Password and Click submit button step 1 is
action Login
2. Enter “Phil Collins” as a Sales Person and Click Submit button
3. Verify the Sales Person was successfully created and Logout
So you may want to choose the following re-usable action keywords:
EnterText, Click, Login, VerifyExists
74. Benefits of Keyword Driven Approach
This Framework addresses the most common
problem with test automation:
Automation Engineers do not have domain
knowledge and the End Users (Subject Matter
Experts/Test Engineers) usually do not have
automation expertise.
When properly implemented and maintained, it
presents a superior ROI because each business
event is designed, automated and maintained as
a discrete entity.
Keywords can then be used to design test
cases, but the design and automation overhead
for the keyword has already been paid.
75. Benefits of Keyword Driven Approach
Reduced the cost and time spent maintaining and
updating tests
The modular structure of keyword-driven testing means
that new tests can easily be created from pre-existing
modules
The test team is capable of entirely automating tests,
even without programming knowledge
Can be easily modified to use with different test tool
Reusability across different projects
Classic Example:
Object Action Data
Textfield (username) Enter Text <username>
76. Keyword/Action Driven Approach
May have an external data source (DB tables, Excel
spreadsheets, XML for data sets) with action keywords
Ste Descriptio
p n Page Action Module Type Object Expected
UserLo .id:=LoginSu .text:=Login
1 Login Home Login gin N/A bmit Successful
Enter New CreateS
Sales alesPers .text:=Sales .value:=Phil
2 Person data on EnterText Field PerName Collins
CreateS
Click alesPers Butto .id:=SubmitS .url:=.*createdSal
3 Submit on Click n alesPer esPerStatus.html
Verify Sales CreateS
Person alesPers .value:=User
Creation onStatus VerifyExist .id:=Creation Created
4 Successful s DIV Status Successfully
77. Hybrid Keyword and Data Driven Approach
Combines the best of both worlds
User defines data sets to drive tests with
User also defines flow control of the test via action
keywords
May have an external data source (DB tables, Excel
spreadsheets, XML for data sets) with action
keywords in addition to generic and test case specific
data sets
82. Model Based Approach
+ abstract tests
+ automatic execution
+ auto regression testing
+ auto design of tests
+ systematic coverage
+ measure coverage of model and
requirements
- modelling overhead
Emerging Approach
84. Summarizing…
Overview of STA Choosing the right tool – The Strategic
History Steps
Myths and Truths Tools Classification
Why Automation Projects Fail? Short List Tools – Functional
What is TA? Short List Tools – Performance
Why TA is needed? Evaluate Vendors
Manual vs. Automated Functional Test Tools - Analysis
Pros & Cons Feasibility Analysis
What to Automate? Automation Framework
What Not to Automate? Where to start?
BEP and ROI What is AF?
BEP Benefits of AF
How does it work?
BEP - Example
Architecture
ROI
Framework Approaches
Classic ROI
Record and Playback
Real ROI Script Based Approach
Benefits of ROI Keyword Driven Approach
ROI Calculator Data Driven Approach
Hybrid Keyword and Data Driven Approach
Tools
Choosing the right tool – The Strategic
Approach