3. • What is Software Test Automation?
Software test automation refers to the activities and efforts that
intend to automate engineering tasks and operations in a software test
process using well-defined strategies and systematic solutions.
The major objectives of software test automation:
- To free engineers from tedious and redundant manual testing
- To speed up a software testing process, and to reduce software testing
cost and time during a software life cycle
- To increase the quality and effectiveness of a software test process by
achieving pre-defined adequate test criteria in a limited schedule
The major key to the success of software automation
--> to reduce manual testing activities and redundant test operations
using a systematic solution to achieve a better testing coverage.
What is Software Test Automation?
4. • Software test automation activities could be performed in three
- Enterprise-oriented test automation, where the major focus of test
automation efforts is to automate an enterprise-oriented test process so
that it could be used and reused to support different product lines and
projects in an organization.
- Product-oriented test automation, where test automation activities are
performed to focus on a specific software product line to support its
related testing activities.
- Project-oriented test automation, where test automation effort is
aimed at a specific project and its test process.
What is Software Test Automation?
5. Different Maturity Levels of Software Test Automation
Level 4: Optimal
Level 2: Repeatable
Level 1: Initial
6. Maturity Level of Software Test Automation
Level 1: Initial
– A software test process at this level provides engineers with systematic
solutions and tools to create, update, and manage all types of software test
information, including test requirements, test cases, test data, test
procedures, test results, test scripts, and problem reports. No systematic
solutions and tools are available to support engineers test design, test
generation, and test executions.
Level 2: Repeatable
– A software test process at this level not only provides engineers with tools
to manage diverse software testing information, but also provides
systematic solutions to execute software tests in a systematic manner.
These solutions allow engineers to use a systematic approach to run tests
and validate test results. However, no systematic solutions and tools are
available to assist test engineers in test design, test generation, and test
7. Maturity Level of Software Test Automation
Level 3: Automatic
– Besides the test management and test execution tools, a software test
process at this level is supported with additional solutions to generate
software tests using systematic methods. They could be useful to generate
black box or white-box software tests. However, no systematic solutions are
available to measure the test coverage of a test process.
Level 4: Optimal
– This is an optimal level of test automation. At this level, systematic
solutions are available to manage test information, execute tests, and
generate tests, and measure test coverage. The primary benefit of achieving
this level is to help engineers understand the current coverage of a test
process, and identify the test coverage issues.
9. Essential Needs of Software Test Automation
A dedicated work force for test automation
The commitment from senior managers and engineers
The dedicated budget and project schedule
A well-defined plan and strategy
Talent engineers and cost-effective testing tools
Maintenance of automated software tests and tools
10. Basic Issues of Software Test Automation
Poor manually performed software test process
Late engagement of software test automation in a software
product life cycle
Unrealistic goals and unreasonable expectations
Lack of good understanding and experience of software test
11. Essential Benefits of Software Test Automation
There are a number of essential benefits from test automation.
They are listed below.
Reduce manual software testing operations and
eliminate redundant testing efforts.
Produce more systematic repeatable software tests,
and generate more consistent testing results.
Execute much more software tests and achieve a
better testing coverage in a very limited schedule.
12. A Software Test Automation Process
Select and Evaluate Available
Software Testing Tools
Develop & Implement
Design Test Automation
Strategies & Solutions
Introduce and Deploy Test
Review and Evaluate
Software Test Automation
13. The Software Test Automation process consists of the following steps:
Step #1: Test automation planning
– This is the initial step in software test automation. The major task here is to
come out a plan that specifies the identified test automation focuses, objectives,
strategies, requirements, schedule and budget.
Step #2: Test automation design
– The primary objective of this step is to draw out the detailed test automation
solutions to achieve the major objectives and meet the given requirements in a
test automation plan.
Step #3: Test tool development
– At this step, the designed test automation solutions are developed and tested as
quality tools and facilities. The key in this step is to make sure that the developed
tools are reliable and reusable with good documentation.
14. The other steps in a test automation process:
Step #4: Test tool deployment
– Similar to commercial tools, the developed test tools and facilities must be
introduced and deployed into a project or onto a product line. At this step, basic
user training is essential, and proper user support is necessary.
Step #5: Review and evaluation
– Whenever a new tool is deployed, a review should be conducted to identify its
issues and limitations, and evaluate its provided features. The review results will
provide valuable feedback to the test automation group for further improvements
16. Scope of Automation in Testing
Automation is the process of evaluating the
aut(application under test) against the specification with the
help of a tool. In this article we are gong to discuss the scope
of automation in testing.
Depending on the nature of testing there are two main branches under
• Functional testing with automation.
• Performance testing with automation.
17. Functional automated testing
Functional automated testing has emerged as a key area in most of the testing processes.
The main area where the functional testing tools are used is for regression test case execution.
Mostly in the agile scrum methodology where frequent releases are happening, it is almost
impossible to execute all the regression test cases manually with the short span of time.
Automation gives a high ROI(Return of Investment) in this area since it is a one time effort for
generating the scripts.
Maintenance to the existing scripts will be considerably less if the automation follows a good
framework which exactly suits the nature of application.
It is advisable to keep at least 60 to 70 % of regression cases to be automated for being adherent to
the timelines of test case execution. Automating test cases is not desirable if it is a onetime testing.
18. Performance Testing with the help of
Performance testing is the process of evaluating the application performance which is being
a critical requirement of every application now days.
Performance testing is almost impossible by manual means.
There are different tools used across organizations for evaluating application performance.
There are different divisions under performance testing based on the nature of testing.
19. Load and Stress testing
Load and Stress testing
Here the application will be tested with the specified load without overloading the application.
But in stress testing, here the intention is to find out the maximum load the application can
Application will be tested with more load than the specified limit in order to find out the break
For load and stress testing, large number of concurrent users is needed and multiple computers are
20. In manual testing it is diffcult and almost impossible to run the severe load for several days to
identify problems in the application.
While using automation the testing software (tool) monitors the CPU and memory usage of each
The tool generates artificial loads by creating virtual users and which will produce a guaranteed
The tools can generate number of virtual users on a single computer. Automation help us to find
out defects as we can run the test using multiple users, multiple environments on several days.
21. Automation is an effective means of eliminating/reducing manual effort
during regression and functional test case execution. Also the chances of
defect escape will be reduced considerably,since human mistakes will not
occur once the script is fully developed. Hence the automation is more
economical considering the fact of time and quality.
23. IT6004 – SOFTWARE TESTING [UNIT – V SNS COLLEGE OF ENGINEERING]
DESIGN AND ARCHITECTURE FOR AUTOMATION
A test case is a set of sequential steps to execute a test operating on a set of predefined inputs to produce
certain expected outputs.
There are two types of test cases namely automated and manual. Test case in this chapter refers to
automated test cases.
A test case can be documented as a set of simple steps, or it could be an assertion statement or a set of
An example of assertion is “Opening a file, which is already opened should fail.” The following table
describes some test cases for the log in example, on how the log in can be tested for different types
25. In the above table the how portion of the test case is called scenarios. What an operation has to do
is a product specific feature and how they are to be run is a framework-specific requirement. When
a set of test cases is combined and associated with a set of scenarios, they are called ͆test suite͇.
27. IT6004 – SOFTWARE TESTING [UNIT – V SNS COLLEGE OF ENGINEERING]
Skills Needed for Automation
The automation of testing is broadly classified into three generations.
First generation – record and playback
Record and playback avoids the repetitive nature of executing tests. Almost all the test tools available in
the market have the record and playback feature. A test engineer records the sequence of actions by
keyboard characters or mouse clicks and those recorded scripts are played back later, in the same order as they
were recorded. When there is frequent change, the record and playback generation of test automation tools may not
be very effective.
Second generation – data – driven
This method helps in developing test scripts that generates the set of input conditions and corresponding
expected output. This enables the tests to be repeated for different input and output conditions. This
generation of automation focuses on input and output conditions using the black box testing approach.
28. IT6004 – SOFTWARE TESTING [UNIT – V SNS COLLEGE OF ENGINEERING]
Third generation action driven
This technique enables a layman to create automated tests; there are no input and expected
output condition required for running the tests. All action that appear on application are
automatically tested based on a generic set of controls defined for automation e input and output
condition are automatically generated and used the scenarios for test execution can be
dynamically changed using the test framework that available in this approach of automation
hence automation in the third generation involves two major aspects ͆test case automation͇
and ͆frame work design͇.
What to Automate, Scope of Automation
The specific requirements can vary from product to product, from situation to situation, from
time to time. The following gives some generic tips for identifying the scope of automation.
29. These types of testing require the test cases to be run from a large number of different machines for an extended period
of time, such as 24 hours, 48 hours, and so on. Test cases belonging to these testing types become the first candidates
Regression tests are repetitive in nature. Given the repetitive nature of the test cases, automation will save significant
time and effort in the long run.
These kinds of tests may require a complex set up and thus required specialized skill, which may not be available on
an ongoing basis. Automating these once, using the expert skill tests, can enable using less-skilled people to run
these tests on an ongoing basis.
Automating areas less prone to change
User interfaces normally go through significant changes during a project. To avoid rework on automated test
cases, proper analysis has to be done to find out the areas of changes to user interfaces, and automate only
those areas that will go through relatively less change. The non- user interface portions of the product can be
automated first. This enables the non-GUI portions of the automation to be reused even when GUI goes through
30. IT6004 – SOFTWARE TESTING [UNIT – V SNS COLLEGE OF ENGINEERING]
Automate tests that pertain to standards
One of the tests that products may have to undergo is compliance to standards. For example, a product providing a JDBC
interface should satisfy the standard JDBC tests. Automating for standards provides a dual advantage. Test suites
developed for standards are not only used for product testing but can also be sold as test tools for the market. Testing for
standards has certain legal requirements. To certify the software, a test suite is developed and handed over to different
companies. This is called ͆certification testing͇and requires perfectly compliant results every time the tests are executed.
Management aspects in automation
Prior to starting automation, adequate effort has to be spent to obtain management commitment. The automated test cases
need to be maintained till the product reaches obsolescence. Since automation involves effort over an extended period
of time, management permissions are only given in phases and part by part. It is important to automate the critical and basic
functionalities of a product first. To achieve this, all test cases need to be prioritized as high, medium, and low, based on
customer expectations. Automation should start from high priority and then over medium and low-priority
31. Design and Architecture for Automation
Design and architecture is an important aspect of automation. As in product development, the design has to represent
all requirements in modules and in the interactions between modules.
In integration testing both internal interfaces and external interfaces have to be captured by design and
architecture. Architecture for test automation involves two major heads: a test infrastructure that covers a test case
database and a defect database or defect repository. Using this infrastructure, the test framework provides a backbone that
ties the selection and execution of test cases.
There are two modules that are external modules to automation – TCDB and defect DB. Manual test cases do not need
any interaction between the framework and TCDB. Test engineers submit the defects for manual test cases. For
automated test cases, the framework can automatically submit the defects to the defect DB during execution. These
external modules can be accessed by any module in automation framework.
32. IT6004 – SOFTWARE TESTING [UNIT – V SNS COLLEGE OF ENGINEERING]
Scenario and configuration file modules
Scenarios are information on ͆how to execute a particular test case. A configuration file
contains a set of variables that are used in automation. A configuration file is important for running the
test cases for various execution conditions and for running the tests for various input and output conditions and
states. The values of variables in this configuration file can be changed dynamically to achieve different
execution input, output and state conditions.
Test case is an object for execution for other modules in the architecture and does not represent any
interaction by itself. A test framework is a module that combines what to execute͇and how they have to be
executed. The test framework is considered the core of automation design. It can be developed by the
organization internally or can be bought from the vendor.
33. Tools and results modules
When a test framework performs its operations, there are a set of tools that may be required. For example,
when test cases are stored as source code files in TCDB, they need to be extracted and compiled by build
tools. In order to run the compiled code, certain runtime tools and utilities may be required.
The results that come out of the test must be stored for future analysis. The history of all the previous
tests run should be recorded and kept as archives. This results help the Test engineer to execute the test cases
compared with the previous test run. The audit of all tests that are run and the related information are stored
in the module of automation. This can also help in selecting test cases for regression runs.
34. IT6004 – SOFTWARE TESTING [UNIT – V SNS COLLEGE OF ENGINEERING]
Report generator and reports /trics modules
Once the results of a test run are available, t.e next step is to prepare the test reports and
metrics. Preparing reports is a complex work and hence it should be part of the automation
design. The periodicity of the reports is different, such as daily, weekly, monthly, and
milestone reports. Having reports of different levels of detail can address the needs of
multiple constituents and thus provide significant returns. The module that takes the
necessary inputs and prepares a formatted report is called a report generator. Once the
results are available, the report generator can generate metrics. All the reports and
metrics that are generated are stored in the reports/metrics module of automation for
future use and analysis.
36. IT6004 – SOFTWARE TESTING
UNIT - V [ UNIT 5 TEST AUTOMATION - SNS COLLEGE OF ENGINEERING]
REQUIREMENTS FOR A TEST TOOL
Generic Requirements for Test Tool/Framework
In the previous section, we described a generic framework for test automation. This section
presents detailed criteria that such a framework and its usage should satisfy.
·No hard coding in the test suite.
·Test case/suite expandability.
·Reuse of code for different types of testing, test cases.
·Automatic setup and cleanup.
·Independent test cases.
·Test case dependency
37. ·Insulating test cases during execution
·Coding standards and directory structure.
·Selective execution of test cases.
·Random execution of test cases.
·Parallel execution of test cases.
·Looping the test cases
·Grouping of test scenarios
·Test case execution based on previous results.
·Remote execution of test cases.
·Automatic archival of test data.23
38. IT6004 – SOFTWARE TESTING
UNIT - V [ UNIT 5 TEST AUTOMATION - SNS COLLEGE OF ENGINEERING]
·Independent of languages
·Probability to different platforms.
Process Model for Automation
The work on automation can go simultaneously with product development and can overlap with multiple
releases of the product. One specific requirement for automation is that the delivery of the automated tests
should be done before the test execution phase so that the deliverables from automation effort can be utilized
for the current release of the product. Test automation life cycle activities bear a strong similarity to product
development activities. Just as product requirements need to be gathered on the product side,
automation requirements too need to be gathered. Similarly, just as product planning, design and coding
are done, so also during test automation are automation planning, design and coding.
39. After introducing testing activities for both the product and above figure includes two parallel sets of activities for
automation, the development and testing separately. When they are put together, it becomes a ͆W͇model. Hence for a
product development involving automation, it will be a good choice to follow the W model to ensure that the quality of
the product as well as the test suite developed meets the expected quality norms.
Selecting a test tool
Having identified the requirements of what to automate, a related question is the choice of an appropriate tool for
automation. Selecting the test tool is an important aspect of test automation for several reasons given below:
1.Free tools are not well supported and get phased out soon.
2.Developing in-house tools take time.
3.Test tools sold by vendors are expensive.
4.Test tools require strong training.
40. IT6004 – SOFTWARE TESTING
UNIT - V [ UNIT 5 TEST AUTOMATION - SNS COLLEGE OF ENGINEERING]
6. Not all test tools run on all platform.
For all the above strong reasons, adequate focus needs to be provided for selecting the right tool for automation.
Criteria for selecting test tools
In the previous section, we looked at some reasons for evaluating the test tools and how requirements
gathering will help. This will change according to context and are different for different companies and
products. We will now look into the broad categories for classifying the criteria. The categories are
41. Meeting requirements
Firstly, there are plenty of tools available in the market, but they do not meet all the requirements of a
given product. Evaluating different tools for different requirements involves significant effort, money
Secondly, test tools are usually one generation behind and may not provide backward or forward
compatibility. Thirdly, test tools may not go through the same amount of evaluation for new
requirements. Finally, a number of test tools cannot differentiate between a product failure and a test failure.
So the test tool must have some intelligence to proactively find out the changes that happened in the product
and accordingly analyze the results.
□Extensibility and customization are important expectations of a test tool.
□ A good number of test tools require their libraries to be liked with product binaries.
42. IT6004 – SOFTWARE TESTING
UNIT - V [ UNIT 5 TEST AUTOMATION - SNS COLLEGE OF ENGINEERING]
□Test tools are not 100% cross platform. When there is an impact analysis of the product on the
network, the first suspect is the test tool and it is uninstalled when such analysis starts.
While test tools require plenty of training, very few vendors provide the training to the required level.
Test tools expect the users to learn new language/scripts and may not use standard
languages/scripts. This increases skill requirements for automation and increases the need for a learning
curve inside the organization.
□Test tools require system upgrades.
□Migration to other test tools difficult
□Deploying tool requires huge planning and effort.
43. Steps for tool selection and deployment
1.Identify your test suite requirements among the generic requirements discussed. Add other
requirements if any.
2.Make sure experiences discussed in previous sections are taken care of.
3.Collect the experiences of other organizations which used similar test tools.
4.Keep a checklist of questions to be asked to the vendors on cost/effort/support.
5.Identify list of tools that meet the above requirements.
6.Evaluate and shortlist one/set of tools and train all test developers on the tool.
7.Deploy the tool across the teams after training all potential users of the tool.
45. Challenges in Automation
• The most important challenge of automation is the management commitment.
• Automation takes time and effort and pays off in the long run.
• Management should have patience and persist with automation.
46. 1) Effective Communication and Collaboration
2) Selecting the Right Tool
3) Demanding Skilled Resources
4) Selecting a Proper Testing Approach
5) High Upfront Investment Cost
Challenges in Automation
48. 1. What is metrics?
2. Why Metrics?
3. Steps for metrics
4. Types of metrics
5. Overview slide
1. Project Metrics
2. Progress Metrics
3. Productivity Metrics
4. Development Metrics
5. Release Metrics
49. What is Metrics
This is the period we noticed excellent profits in the organization …and…
Boss, you were on vacation that period!
1. Set of data is called information and set of information combined to provide a
perspective is called Metrics.
2. A quantitative measure to explain at what degree an attribute of testing or product
quality or process has performed is called Metrics.
3. Effort is the actual time that is spent on a particular activity or a phase. “Elapsed
days” is the difference between start of an activity to completion of the activity.
3. Measurement is an unit used by metrics (e.g Effort, elapsed days, number of defects
…etc). A metric typically uses one of more measurements
51. Why Metrics?
1. How do you determine quality and progress of testing?
2. How much testing is completed?
3. How much more time is needed for release?
4. How much time needed to fix defects?
5. How many Days needed for release?
6. How many defects that will be reported by customers?
7. Do you know how to prevent defects rather than finding and fixing them?
Do you have answers?
52. Why Metrics for QA?
1. Testing is penultimate cycle of product release --- Determining quality and progress of testing thus
is very important
2. How much testing is completed can be measured if you know how much total testing is needed
3. How much more time is needed for release (e.g) Days needed to complete testing = total test
cases yet to be executed / test case execution productivity
4. How much time needed to fix defects (e.g) The defect trend gives a rough estimate of defects that
will come in future. Metrics helps in predicting the number of defects that can be found in future
test cycles (e.g) Total days needed for defect fixes = (Outstanding defects yet to be fixed + Defects
that can be found in future test cycles) / defect fixing capability
53. Steps for metrics
Step 1: Identify what measurements are important
Step 2: Define granularity of measurements ; Granularity depends on data drilling. Example
Tester: We found 100 more defects in this test pass compared to the previous one
Manager: What aspect of the product testing produced more defects?
Tester: Functionality aspect produced 60 defects out of 100
Manager: Good, what are the components in the product that produced more functional defects?
Tester: “Installation” component produced 40 out of those 60
Manager: What particular feature produced that many defects?
Tester: The data migration involving different schema produced 35 out of those 40 defects…….
54. Steps for metrics
•Step 3: Decide on periodicity of metrics
•Step 4: Analyze metrics and take action items for both positives and
•Step 5-n: Track action items from metrics
55. Types of metrics
Project metrics: The set of metrics which indicate how the project is planned and executed
Progress metrics: The set of metrics to indicate how different activities of the project are progressing. The
activities include both development and testing activities. Since the focus of this training is testing, only
those metrics applicable to testing are discussed.
Productivity metrics: The set of metrics that takes into account various productivity numbers that can be
collected and used for planning and tracking the testing activities.
85. 5.7 Test Metrics and Measurement
“STANDARDS OF MEASUREMENT”
Metric is a quantitative measure of the degree to which a system, system component, or
process possesses a given attribute.
"Software testing metrics - Improves the efficiency and effectiveness of a software testing process."
86. What is software test measurement?
Quantitative indication of extent, capacity, dimension, amount or size of some attribute of a process or
Why do Test Metrics?
"We cannot improve what we cannot measure"
Take decision for next phase of activities
Evidence of the claim or prediction
Understand the type of improvement required
Take decision or process or technology change
89. 5.7.1 TYPES OF METRICS
•Process Metrics: It can be used to improve the process efficiency of the SDLC ( Software Development Life Cycle)
•Product Metrics: It deals with the quality of the software product
•Project Metrics: It can be used to measure the efficiency of a project team or any testing tools being used by the team
90. IDENTIFICATION OF TEST METRICS
Fix the target audience for the metric preparation.
Define the goal for metrics.
Introduce all the relevant metrics based on project needs.
Analyze the cost benefits aspect of each metrics and the project lifestyle phase in which it results into the
91. 220.127.116.11 Manual Test Metrics
Raw data collected by Test Analyst during the test case development and execution (# of test cases executed,
# of test cases).
Derived from the data collected in base metrics.
Followed by the test manager for test reporting purpose (% Complete, % Test Coverage).
92. OTHER IMPORTANT METRICS:
Test case execution productivity metrics
Test case preparation productivity metrics
Defects by priority
Defects by severity
Defect slippage ratio
94. 5.7.2 How to calculate test metric
Percentage test cases executed= (No of test cases executed/ Total no of test cases written) X 100
Percentage test cases not executed= (No of test cases not executed/ Total no of test cases written) X 100
Percentage test cases passed= (No of test cases passed / Total no of test cases executed) X 100
95. Percentage test cases failed = (No of test cases failed / Total no of test cases executed) X 100
Percentage test cases blocked= (No of test cases blocked / Total no of test cases executed) X 100
Defect density = No of Defects identified/size
30/5 = 6%
Defect density removal(DRE) = No of Defects found during QA testing/(No of Defects found during QA testing
+ No of defcts found by end user))*100
DRE = [100/(100+40)]*100=71%
96. Defect leakage = No of Defects found in UAT / Nof of defects found in QA testing )*100
(40/100)*100 = 40%
% of critical defects = No of critical defects identifed / total no of defects identifed * 100
% of high defects = No of high defects identifed / total no of defects identifed * 100
% of medium defects = No of medium defects identifed / total no of defects identifed * 100
6 /30*100= 20%
97. % of low defects = No of low defects identifed / total no of defects identifed * 100
8 /30*100= 27%