מנהלים רבים וארגונים רבים מיישמים אוטומציה בתהליך הבדיקות שלהם אבל עדיין מרגישים שההחזר על ההשקעה נמוך ואף שלילי. מחקרים רבים מראים כי הבעיה נובעת מחוסר תיאום ציפיות, זיהוי לא נכון של הבעיות שהכלים באים לפתור, בחירת כלי לא מתאים ותהליך הטמעה שגוי. מתודולוגיית TMM (Testing tools management) באה לתת מענה בדיוק לבעיות שהוצגו. המתודולוגיה כוללת הגדרת השלבים השונים בפרויקט אוטומציה, החל מהגדרת הבעיה, דרך בחירת הכלי, בחינת הכלי, הטמעה ומדידת האפקטיביות שלו לכל אורך הפרויקט

1. Testing Tools Management Methodology
How to Manage
your Testing Automation Project
Lior Katz - Testing Automation & performance leader,
Tact testing (Matrix)
Jul 14th 2010
1 ©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

2. Who am I ?
2 Footer Goes Here

4. “Test Automation is probably the most
misunderstood concept in the field of
software testing. The automation tool
implementation in testing is a costly process
and requires careful planning and execution.
Test Automation has to be treated like a full
blown software development project as the
final key deliverable, is in fact a software
code ”
–Software test practitioners
4 Footer Goes Here

5. “In general, the efforts to implement
Automated Tools and realize ROI have been
marginally successful. One of the main
reasons for this failure has been the lack of
planning and inadequate funding needed to
deliver key initial services around the
deployment of tools,
training, and mentoring of resources by the
purchasing parties. Many IT managers still
believe that the tools will and should work by
themselves.”
2009 WORLD QUALITY REPORT
5 Footer Goes Here

6. Testing Automation projects
Testing Automation project life cycle
1 Purchase the tool 2 Recruit developers 3 Understand ROI 4 Fail to implement
Should be - Testing Automation projects life cycle
Analyze the systems Recruit developers Follow TTM
1 2 3 Understand ROI 4
Purchase the tools and managers methodology
6 Footer Goes Here

7. Testing Tools Management methodology
7 Footer Goes Here

8. TTM methodology - motivation
A methodology developed by myself after years
of experience, to understand the market need and
the breaking points of testing automation projects
and load testing projects.
8 Footer Goes Here

9. Why Do We Fail?
We have the best people and the best tools
but still most projects are failing
management expectations
9 Footer Goes Here

10. Why Do We Fail
How to succeed with your testing automation project using the TTM
– management support (not enough)
– high expectations for immediate ROI
– Unrealistic expectations from a tool
– Underestimating the time, cost and effort for the initial introduction and implementation
of a tool (e.g. training, external expertise and changes to the testing process).
– Underestimating the effort required to maintain the test scripts and tool updates
– Over-reliance on the tool results
– robust scripting
–Understand the application and the development process
– How to divide our automation project in order to succeed?
A MUST is to decide first regarding Testing automation packages progress
(Sanity, Mini Regression, Regression, divide by modules)
10 Footer Goes Here

11. TTM process - 4 steps
1. Define the project scope
2. Follow up
3. Testing Automation methodology
4. closer
11 Footer Goes Here

12. 1. Define the Project Scope
– High Level Design (Test Plan)
The document contains the Set we are about to automate, the concept of
automation, verification points, parameters, etc…
12 Footer Goes Here

13. 1. Define the Project Scope
– Design (Test cases QC)
Test cases developed in QC cover all required area in the application
13 Footer Goes Here

14. 1. Define the Project Scope
– Estimation module
A tool can assist to understand how many working days we need for the
project, include all required steps:
• Testing priority
• All Automation steps
• Risk factor
14 Footer Goes Here

15. 1. Define the Project Scope
– Work plan
Following the estimation module solution the exact work plan by days
15 Footer Goes Here

16. 2. Follow Up - Test Cases list
16 Footer Goes Here

17. 2. Follow Up - Work plan
17 Footer Goes Here

18. 3. Testing Automation Methodology
– QTP Standard and Guidelines
Documents introduces basic guidelines and code methods for scripting via
QTP. The document describes all acceptable practices to develop in QTP:
18 Footer Goes Here

19. 3. Testing Automation Methodology
– Code inspection
All of the inspectors to reach consensus on a work product and approve it for
use in the project
19 Footer Goes Here

20. 4. Closer activities
– Final Report - A document contains all project steps include:
– Test case list
– Testing automation coverage
– Functions
– Folder where script
– How to execute the SET
– Result example
20 Footer Goes Here

21. 4. Closer activities
– Code Freeze
A process to save all developed code under one place. (With a limited access)
21 Footer Goes Here

22. Testing Automation Stages
– Sanity SET
Set of ~50 QTP scripts, verify that new build (version) ready for massive testing process
– Quick ROI Developments
Packages can retrieve the investment immediately (e.g. API, data driven, data inflation)
– Regression by modules - Do Not to develop all regression tests in one
SET, “Before moving forward with the development stage, we need to
analyze our system and divide it by modules, for each module to
develop it’s own automation SET”
– Data Inflation
• Using our GUI interface can help with massive data changes
22 Footer Goes Here

23. TTM methodology - ROI example
– Each execution was reduced by ~20 working days
– No. of execution per version - ~7
– Total = a reduction of 140 working days per version
– One working day = ~225$
Saved for the organization ~31500$ per
version
23 Footer Goes Here

24. Thank you all!!!
24 Footer Goes Here