1. CHAPTER 10
An Introduction to
Automated Software Testing

2. CHAPTER 10
OUTLINE
• What is Test Automation?
• The Benefits and Pitfalls of Test Automation
• When to Use Test Automation
• Test Automation Production Process
• The Evolution of Test Automation

3. CHAPTER 10
OBJECTIVES
• Present an overview of test automation engineering
• Understand the pros and cons of test automation
• Test automation strategies, tactics and implementation

4. An Introduction to
CHAPTER 10.1 Automated Testing
10.1 What is Test Automation?
10.2 The Benefits and Pitfalls of Test Automation
10.3 When to Use Test Automation
10.4 Test Automation Production Process
10.5 The Evolution of Test Automation

5. What is Test Automation?
CHAPTER 10.1
• Using computer time to execute of tests.
• Commonly, test automation involves automating existing and
sometimes, new tests that otherwise, will be executed by a
human tester

6. Why Automate Testing?
• People might say, “Automation will help us release a
higher quality product on time.”
– What does that mean?
• What’s the goal of instituting an Automation
program?
– What problem are we trying to solve?

7. Why Automate Testing?
– Not enough test time in schedule for repetitive tasks
– Very repetitive tasks
– Too many releases to test
– Better use of qualified in-house testing resources
– Reduce cost of testers
– Can’t test manually
– Optimize tester time because there is not enough time for
mundane tests
– Human tester can focus on testing that requires complex analysis
– Save money
– Save time

8. Why Automate Testing?
– If I automate testing, what manual testing will I lose? How
many bugs might I lose and when?
Remember: Bugs are more expensive to fix later.

9. Why Automate Testing?
– Often, if the manual test team also does automation, it is the best
or most technically talented testers who script.
– The early part of application testing is when the most talented testers are
needed.
• This is particularly true with script maintenance. Some talented
person on the main product team is often in charge of maintaining
the automation project.
– Will we lose finding good early bugs because of this?
– Testing cannot be an “on the side” job. You can’t write good
scripts while you wait to get a build or have spare time!

10. Goal of Automation
Sometimes, the goal of a test automation program can be
Good-Enough Automation. Just like with the products we sell,
quality is a combination of reliability, features and timeliness.
If your goal of automation is tightening the testing cycle. AND
you can also improve the test battery, you have done a good
job.

11. Manual vs.
CHAPTER 10.1 Automation Testing
Manual Testing Automation Testing
Test execution time  Test execution time 
Regression test coverage Regression test coverage 

Responsive and flexible Relatively inflexible
Human variation Machine consistency
No maintenance needed High maintenance needed
Immediate dividend paid Long-term dividend
Not cost effective for high Not cost effective for low
volume regression volume regression

12. An Introduction to
CHAPTER 10.2 Automated Testing
10.1 What is Test Automation?
10.2 The Benefits and Pitfalls of Test Automation
10.3 When to Use Test Automation
10.4 Test Automation Production Process
10.5 The Evolution of Test Automation

13. Test Automation
CHAPTER 10.2 Benefits
• Run automated tests unattended (overnight)
• Reliable, consistent repetitive testing
• Increased speed in test execution
• Improves quality, increased test coverage
• Reduces costs and time for regression testing
• Optimize the use of test resources to focus on bug findings
• Reusable and scalable if architected effectively
• Execution of tests that can’t be done manually
• Performance testing to assess software stability, concurrency,
scalability….

14. Test Automation
CHAPTER 10.2 Pitfalls
• Uncertainty and lack of control.
• Poor scalability and maintainability.
– Too sensitive to changes
– Poor reusability
• Low test automation coverage
– Too much programming
• Poor methods and disappointing quality of tests
– Focus of tools instead production volume
– Focus of making the tests run instead of quality of the tests
• Technology vs. people issues
– Gaps in platform technology support
– Gaps in human resource equipped with test automation skills

15. An Introduction to
CHAPTER 10.3 Automated Testing
10.1 What is Test Automation?
10.2 The Benefits and Pitfalls of Test Automation
10.3 When to Use Test Automation
10.4 Test Automation Production Process
10.5 The Evolution of Test Automation

16. Which Test Cases
CHAPTER 10.3 Can’t be Automated?
• Usability testing
"How easy is the application to use?“
• One-time testing
• “ASAP” testing
“We need to test NOW!”
• Ad hoc/random testing
Based on intuition and knowledge of application
• Tests without predictable results

17. The Criteria to
CHAPTER 10.3 Apply Test Automation
• Customer requests
• Application should be stable
• Tests that need to be run for every build of the application (sanity
check, regression test)
• Tests that use multiple data values for the same actions (data driven
tests)
• Tests that require detailed information from application internals
(memory, CPU)
• Tests that require considerable amount of time to perform
manually
• Stress, load or performance testing

18. An Introduction to
CHAPTER 10.3 Automated Testing
10.1 What is Test Automation?
10.2 The Benefits and Pitfalls of Test Automation
10.3 When to Use Test Automation
10.4 Test Automation Production Process
10.5 The Evolution of Test Automation

19. Test Automation
CHAPTER 10.4 Production Process
Repeat steps
Wait for
Input user Check AUT until all to-be-
processing to
actions function outputs executed tests
complete
are completed
1 2 3 4
Manual
Synchronize
Generate script playback
Run tests or
automated to application Add checks
suites of tests
scripts output from
1 each step 2 3 4
Automated

20. An Introduction to
CHAPTER 10.5 Automated Testing
10.1 What is Test Automation?
10.2 The Benefits and Pitfalls of Test Automation
10.3 When to Use Test Automation
10.4 Test Automation Production Process
10.5 The Evolution of Test Automation

21. A Brief History
CHAPTER 10.5 of Test Automation
Software Test Automation has evolved through several generations,
with many improvements along the way:
1st Generation: Record and Playback
2nd Generation: Scripting
3rd Generation: Data-driven
4th Generation: Keyword-driven

22. Record & Playback
CHAPTER 10.6
 The automation tool records the steps of the user executing a
test, and the tester configures some checks.
 The automation tool generates a script based on the recorded
actions.
 The automation tool performs a playback of the recorded test.
 Even small changes to the application under test require manual
re-recording of the test.
→ This first generation is neither efficient nor scalable.

23. Record/Playback
Recorded test • Actions
• Checks
scripts
• One time recording of Playback
actions and checks
• Multiple time playback
Test Tool Log
(Invisible to the
target system)
Record Playback • Actions
• Checks
• Results
Test data entered by Target
testers or users System/AUT

24. EXAMPLE
A Recorded Script
select window "Logon"
enter text "username", "administrator"
enter text "password", "testonly"
push button "Ok"
select window "Main"
push button "New Customer"
expect window "Customer Information"
select field "Name"
type "Jones"
select field "First Name"
type "John"
select field "Address"
type "54321 Space Drive"
.
.
.

25. EXAMPLE
A Recorded Script
CHAPTER 10.6
• Dependent on screen coordinates
# Analog Recording
move_locator_track (1); mouse movement
mtype ("<T55> <kLeft>-<kLeft>+"); mouse click
type ("<t3>output14" );
move_locator_track (2);
mtype ("<T35><kLeft>-<kLeft>+");
output14
enter text pause before
from mouse click
keyboard

26. R/P Pros and Cons
• Pros
– A useful way to learn about the testing tool and
automation
• Cons
– High maintenance cost dealing with changes in the target
system of AUT
– Poor accessibility to test cases
– Greatly affected by the working system or environment
– Only suitable for GUI systems

27. Scripting Approach
• Automation is regarded as a programming activity
• Parameterize hard-coded values
• Separate data from code by moving variables to include files
• Create utility functions to be shared
• Produce and maintain like any other software
• Train black-box test engineers to run the scripts

28. Scripting Approach
• Pros
– Reusability of scripts for common tasks
– Improved maintainability
• Cons
– Poor accessibility to test cases because tests are
programmed or scripted
– Greatly affected by a working system or environment
– Test implementation must be done by programmers
– Translation of test cases—3000 tests mean 3000
automated scripts
– Manageability is still an issue due lack of visibility

29. EXAMPLE
Function EnterCustomerJones
Logon
Press "New Customer"
Enter Field "Name", "Jones"
Enter Field "First Name", "John"
Enter Field "Address", "54321 Space Drive"
...
Logoff
End Function
Function OrderProduct
Logon
Press "New Order"
Enter Field "Product", “OurProduct 1.0"
Enter Field "Amount", "35"
Enter Field "Delivery", "asap"
...
LogOff
End Function
...

30. Scripting –
Example Script
CHAPTER 10.6
Silk Test Script:

31. (Scripting) Data-Driven
Approach
• Take advantage of tester’s familiarity with test case design and
creation using tables and matrices
• Accommodate localization projects
• Recognize the importance of patterns in test cases
• Enable testers to catalog test cases with spreadsheets in
software such as Excel
• Enable testers to specify expected results in spreadsheets

32. EXAMPLE
GOODMORNING
goodmorning
GoodMorning
Input Data

33. EXAMPLE
• File with test data: • Text file used in the test:
nr case whole pattern matches
goodmorning
1 off off GOODMORNING 4
2 off on GOODMORNING 3
GOODMORNING
3 on off GOODMORNING 1 GoodMorning
4 on on GOODMORNING 1 GoodMorningVietnam
5 off off goodmorning 4
6 off on goodmorning 3 • Data driven script:
7 on off goodmorning 1 for each line in the file do
8 on on goodmorning 1 open the find dialog
9 off off GoodMorning 4 read a line from the file
10 off on GoodMorning 3 use the values to fill dialog
11 on off GoodMorning 2 press find button
12 on on GoodMorning 1 check amount of matches
close the dialog

34. (Scripting) Data-Driven
Approach
• Pros
– Separation of action and data enables tests to receive
combinations of test data
– Test data can be stored in a separate file or spreadsheet for better
maintainability
– Improved reusability and maintainability
• Cons
– Actions containing business logic are in automated scripts
– Maintainability of changes in business logics as well as interface is
still poor
– Poor visibility because accessibility to the tests is still poor
– Not scalable (e.g., reaching 95% of automation)

35. Test Automation
Challenges
• Too much programming
• Response to rate of changes (software)
• Low reusability/scalability
• Too many platforms/technologies
• Too expensive if the returned value is low

36. Keyword-driven
CHAPTER 10.6
 Tests are broken down into a series of actions representing the
business processes of the SUT, such as ‘enter client’, ‘transfer’, and
‘check balance’.
 Individual actions are automated, not the tests.
 The tests are stored externally from the test scripts, typically in a
spreadsheet.
 Keyword-based test automation’s elegant, structured approach
simplifies the work even further, reducing the time and expense of test
design, automation, and execution by allowing team members to focus
on their areas of individual expertise.
 Non-technical testers and business analysts can develop executable
test automation from keywords representing recognizable end-user
actions.

37. How Does ABT
(Action-keyword) Work?
 Test Cases are written or described through a series of Test Lines using
parameterized and reusable Action keywords
Action’s Parameters
Action keyword Number Name Description
Test Line add product 12345678 Hammer For pounding nails
 Actions are created and communicate with target systems and software
applications including the application under test (AUT) through one or more
supported Interfaces to carry out intended tasks
Note:
For now, think of an Action as a verb or something that you do
 Information about Interfaces such as Windows, Web or Java graphical user
interface (GUI) or command-line interface (CLM) are later defined to enable
actions to send inputs to and receive outputs from the target system and/or
AUT

38. Overcome Challenges w/ Action Based Testing
• Separation of action, interface, logic and data
• Test data can go in a separate file or spreadsheet
• Test logic can go in a separate file or spreadsheet
• Interface can go in a separate file or spreadsheet
• Action can also go in a separate file or spreadsheet
• Action can be reused

39. Automation Approach
Comparison
CHAPTER 10.6
Workload Allocation
Approach Test Data
Automation Test Case
Construction /
Engineering Production
Selection
Record &
N/A Test Engineer N/A
Playback
Automation Automation
Scripting Engineer Engineer
N/A
Automation Automation
Data-driven Engineer Engineer
Test Engineer
Automation
Keyword-driven Engineer
Test Engineer Test Engineer
Note: Programming skill is required for Automation Engineer

40. Examples of Test
CHAPTER 10.6 Automation Tools
LogiGear is the producer of TestArchitect™, a functional automated testing tool
designed to help achieve the most efficiency out of action-keyword test
automation. Examples of other commercial tool vendors include:
Automation HP IBM Others
Tool
Functional WinRunner Functional Tester SilkTest
QuickTestPro Rational Robot TestPartner
TestComplete
Performance LoadRunner Performance Tester SilkPerformer
AstraLoad Test WebLoad
OpenSTA
Management Test Director Rational TestManager SilkCentral Test
Quality Center Manager
Memory Purify JProbe Profiler

41. CHAPTER 10
SUMMARY
• Test Automation is a method that uses of software to control
the execution of tests.
• Commonly, test automation involves automating a manual
process currently in.
• Test Automation has benefits such as…However, it still has
some pitfalls such as…
• A effective strategy of test automation includes several steps…
• There are many popular automation tools. Depend on the
AUT, we choose the suitable automation testing tool.

42. CHAPTER 10
QUIZ
• What are the advantages and disadvantages of test
automation?
• What are the advantages and disadvantages of test
automation generations?