1. ISTQB CTAL Test Manager (TM)
A Subject Overview of the
ISTQB CTAL TM Certification
Petro Porchuk
Lohika 2010
Copyright Petro Porchuk 2010. Lohika
2. My Goal
Copyright Petro Porchuk 2010. Lohika
Present topics due to the Syllabus CTAL
90% of the info have the ISTQB colour
Give some references
10% - Express my own Vision
http://istqb.org/download/attachments/2326555/CTAL_Syllabus_V_2007.pdf
3. My Main Source
Copyright Petro Porchuk 2010. Lohika
Software Testing Practice:
Test Management:
A Study Guide for the Certified
Tester Exam Istqb Advanced
Level
Andreas Spillner (Author)
http://www.amazon.com/Software-Testing-Practice-Management-Certified/dp/193395213X/ref=sr_1_6?ie=UTF8&s=books&qid=1275570124&sr=8-6
4. The ISTQB Framework, May 2010
http://istqb.org/display/ISTQB/Certification?atl_token=CrArtYjmmm
Copyright Petro Porchuk 2010. Lohika
5. CTAL TM – Part 1
Part 1
1. Introduction
2. Test Process and Test Tools
3. Testing in the software Life Cycle
4. Test Policy and Test Handbook
5. The Test Plan
6. Test Control
7. Assessing and Improving the Development and Test Processes
Part 2
8. Deviation Management
9. Risk Management and Risk-Oriented Testing
10. Staff Qualification and Skills
11. Test Metrics
12. Selecting and Implementing Test Tools
13. Standards Copyright Petro Porchuk 2010. Lohika
6. (1) The Introduction
Copyright Petro Porchuk 2010. Lohika
TL should be:
Technically Skilled
Communication and
Language Skilled
Good Analyst
Pro-Active
Brave Organizer
http://www.lestnizza.com.ua/upload/image/it_range.jpg
7. Copyright Petro Porchuk 2010. Lohika
(2)
(2) Test process and Test Tools
Test process and Test Tools
The Fundamental Test Process
Different Types of Test Tools
Tools for Management and Test Control
Tools for Test Data and Test Script Specific.
Tools for Static Testing
Tools for Dynamic Testing
9. 2.2. The Test Tools
CASE Tools (Computer Aided Software Engineering)
CAST Tools (Computer Aided Software Testing)
Tools for Management and Test Control
(Planning, Defect Management, Requirement Management, Test Spec. Management)
Tools for Test Data and Test Script Specific
(Test Data Generators, Test Generators)
Tools for Static Testing
(Static analizers, model analize spesifications, reviews tools)
Tools for Dynamic Testing
(Debugger, drivers, stubs, test robots, L/P test tools, monitors)
Copyright Petro Porchuk 2010. Lohika
10. ”A Fool with a Tool is still a Fool!”
Copyright Petro Porchuk 2010. Lohika
http://www.sungsblog.com/labels/world.php
11. 2.3. Summary
Copyright Petro Porchuk 2010. Lohika
✔ Testing must be divided into individual process steps
✔ For each phase of the Test Process Tools are available
✔ The use of the Test Tools is only of advantage if a Test
Process is controlled and defined process
✔ ”A Fool with a Tool is still a Fool”
12. We All Do Work Process Oriented
We All Do Work Process Oriented
Copyright Petro Porchuk 2010. Lohika
13. Copyright Petro Porchuk 2010. Lohika
(3)
(3) Testing in the Software Life Cycle
Testing in the Software Life Cycle
Test and Development Process
The General V-Model
The W-Model
Rational Unified Process (RUP)
V-Model XT
Extreme Programming
Rapid Application Development (RAD)
Dynamic System Development Method (DSDM)
14. 3.1. Test and Development Process
Copyright Petro Porchuk 2010. Lohika
QA Process should have strong interaction with the
Development Process
Reaction to the Requirement Change
Configuration Management
Project Management
The General Test Process should be adjusted to the
specific activities applied Development Processes
15. 3.2. The General V-Model
http://www.testingexcellence.com/v-model/
Copyright Petro Porchuk 2010. Lohika
16. 3.3. The V-Model for QA
Copyright Petro Porchuk 2010. Lohika
Pros:
Emphasizes Test
Levels
Verification
Validation
Cons:
Preparatory Test
Activities are missing
18. 3.5. The W-Model for QA
Copyright Petro Porchuk 2010. Lohika
Pros:
Preparatory Test
Activities are here
Concurrency of Test
and Dev. Processes
PREviews
Debugging
Cons:
Needs several Test
Managers for large
projects
Refers only to a
System Development,
not Process Oriented
19. 3.6. The Rational Unified Process
http://www.wittmannclan.de/ptr/cs/slcycles.html
Copyright Petro Porchuk 2010. Lohika
20. 3.7. QA in RUP
Copyright Petro Porchuk 2010. Lohika
Pros:
Test Activities begin
early and accompany
other processes
Cons:
Which Test Activities
to be eprformed when
is not made clear
21. 3.8. V-Model XT (eXtreme Tailoring)
https://sosa.ucsd.edu/teaching/cse294/fall2004/pres/vmxt.htm
Copyright Petro Porchuk 2010. Lohika
The current version of the V-Model is
the V-Model 97. Since 1997, no
changes or improvements were made.
In 2002, a project was initiated to
redesign and improve the existing
process model. The resulting V-Model
XT should reflect new standards and
technologies; it also should expose
significantly enhanced quality
properties such as usability,
adaptability, changeability and
scalability.
Available since 2005.
22. 3.9. QA in V-Model XT
Copyright Petro Porchuk 2010. Lohika
Pros:
Tailoring
Particular attention to
Customer-Supplier
communication
Quality Management
Cons:
No Test Manager
role, but Quality
Manager
23. 3.10. The Extreme Programming
Copyright Petro Porchuk 2010. Lohika
http://www.extremeprogramming.org/map/project.html
24. 3.11. QA in XP
Copyright Petro Porchuk 2010. Lohika
Pros:
Testing is at the
center of the
development effort
(Test First)
Advanced Unit
Testing
Cons:
Missing systematic
Test Specification
creation
QA misses
Aceptance Testing,
it's been put on a
customer
25. 3.12. Rapid Application Development
Copyright Petro Porchuk 2010. Lohika
http://www.antrix.com/services/service_list/11_validation_computer_systems.htm
26. 3.13. QA in RAD
Copyright Petro Porchuk 2010. Lohika
Pros:
Extensive use of
Tools
Invloving Customer to
evaluate
Cons:
Roles of Testing or
Test Management are
not defined, there is
”SWAT Team”
instead.
”SWAT” = Skilled Workers and Advanced Tools
27. 3.14. Dynamic System Develpment Method
Copyright Petro Porchuk 2010. Lohika
DSDM = SCRUM
SCRUM = RAD + (precise process steps and roles)
Focus on Teamwork
Close Cooperation with a Customer
Iterative System Development
Time Boxing
”MoSCoW” Principle
28. 3.15. QA in SCRUM
Copyright Petro Porchuk 2010. Lohika
Pros:
Team driven
evolutionary process
Cons:
Test Manager role is
not here.
Tester is responsible
for component tests
mostly.
29. (3) Summary
Copyright Petro Porchuk 2010. Lohika
✔TM should adopt Test Process to the chosen
Development Model
✔TM should make sure the docs are available to
have enough time for the test preparation
✔TM should make sure Test Activities start early
in the Process
✔TM should make sure the needed Test Levels
are planned and executed
30. (4)
(4) Test Policy and Test Handbook
Test Policy and Test Handbook
Copyright Petro Porchuk 2010. Lohika
Quality Policy and Test Policy
Bring the Test Policy to Life
Test Handbook
31. 4.1. Quality Policy and Test Policy
4.1. Quality Policy and Test Policy
Copyright Petro Porchuk 2010. Lohika
Quality Policy
Expresses the particular importance that an organization
attaches to ”quality” and the demands it makes on the quality of
it's products, services and processes.
Test Policy
A high level document describing the principles, approach and
major objectives of the organization regarding testing.
Test Handbook
A concretization of the Test Policy
http://istqb.org/download/attachments/2326555/ISTQB+Glossary+of+Testing+Terms+2+1.pdf
32. 4.3. Bring the Test Policy to Life
4.3. Bring the Test Policy to Life
Copyright Petro Porchuk 2010. Lohika
Relevance to Business Objective
Be Realistic
Adequate Maturity
Measurability
Liveliness
34. A Question
A Question
Copyright Petro Porchuk 2010. Lohika
What is the most Common Question for the QA
interview
?
http://profiles.friendster.com/98111238
35. A Question
A Question
Copyright Petro Porchuk 2010. Lohika
What is the most Common Question for the QA
interview
?
What is the Test Plan and
what does it consist of?
http://profiles.friendster.com/98111238
36. (5)
(5) The Test Plan
The Test Plan
Copyright Petro Porchuk 2010. Lohika
General Test Plan Structure
Defining a Test Strategy
Test Effort Estimation
Flat Models
Detailed Models baced on Test Activities
Models baced on Functional Volume
Organization of Test Teams and Test Levels
37. 5.1. The General Test Plan Structure
5.1. The General Test Plan Structure
Copyright Petro Porchuk 2010. Lohika
Chapter According to IEEE 829
1. Test plan identifier
2. Introduction
3. Test items
4. Features to be tested
5. Features not to be tested
6. Approach
7. Item pass/fail criteria (test exit criteria)
8. Suspension criteria and resumption requirements
9. Test deliverables
10. Testing tasks
11. Environmental needs
12. Responsibilities
13. Staffing and training needs
14. Schedule
15. Risk and contingencies
16. Approvals
38. 5.2. Defining the Test Strategy
5.2. Defining the Test Strategy
Copyright Petro Porchuk 2010. Lohika
39. 5.3. Test Effort Estimation – Flat Models
5.3. Test Effort Estimation – Flat Models
Copyright Petro Porchuk 2010. Lohika
Adding a percentage flat rate to the
development effort
QA Effort = 0.5*(DEV Effort)
Assuming development estimation is accurate
Take into an account experience from similar
projects
It makes sence to use several sources
But:
Cannot be used for big projects
40. 5.4. Test Effort Estimation – Detailed Models
5.4. Test Effort Estimation – Detailed Models
Copyright Petro Porchuk 2010. Lohika
Concatenation of individual activities and
different estimation methods must be used for
each activity. Consider:
Manpower
Time
HW and SW for the Test Environment
Testware (test automation tools, etc)
Different influencing factors
Keep estimated tasks small
Involve the team in estimation
Use experience
41. 5.5.1. Test Effort Estimation – Model Based
5.5.1. Test Effort Estimation – Model Based
Copyright Petro Porchuk 2010. Lohika
Estimation of the Scope or Functionality of the
test objects and establishing a mathematical
model
FPA (Function Point Analysis)
Count the number of functions to be developed
Add the non-functional values
(14*(1..5)+0.65 * func)
Transform to man*hours
The Test Effort is a result of estimates from function
points.
But: Used only for simply numbers (of test cases)
42. 5.5.2. Test Effort Estimation – Model Based
5.5.2. Test Effort Estimation – Model Based
Copyright Petro Porchuk 2010. Lohika
Test Point Analysis Uses Quality Atributes (ISO 9126)
Dynamic Test Points (FP; Quality Attributes; Function priority,
compexity, influence etc) Are multiplied with each other per
function and summed up. This reflects the dynamic test efforts.
Static Test Points (flexibility, testability, tracebility) The total
number of FP is multiplied by a factor and added up. Static test
efforts
Dynamic and Static test points are summed up and subsequently
weighted with environment factors (docs, testware, tools,
organization culture etc)
The result – is the number of test hours required
43. (5) Summary
Copyright Petro Porchuk 2010. Lohika
✔ The Test Plan is the concrete application of the Test
Handbook, mapping the strategic requirement of the
Test Policy into the Project
✔ Essential Test Plan components are:
Test Strategy
Test Effort Estimation
Test Organisation
✔ For Test Effort Estimation may be applied:
Intuition, experience, detailed itemization, formulas
✔ The WORSE estimation is NO ESTIMATION
44. (6)
(6) Test Control
Test Control
Copyright Petro Porchuk 2010. Lohika
✔Initializing the Test Tasks
✔Monitoring the Test Process
✔Reacting to Test Results
✔Reacting to Changed Circumstances
✔Evaluating the Test Completion
✔Test Report
45. Standish Group Chaos Report 2009
Standish Group Chaos Report 2009
Copyright Petro Porchuk 2010. Lohika
Successful Projects
32%
Late, Overbudget, Miss features
44%
Failed Projects
24%
http://www.standishgroup.com/newsroom/chaos_2009.php
"These numbers represent a downtick in the success rates from the previous study, as
well as a significant increase in the number of failures", says Jim Crear, Standish Group
CIO, "They are low point in the last five study periods. This year's results represent the
highest failure rate in over a decade"
46. How to Fix This?
How to Fix This?
Copyright Petro Porchuk 2010. Lohika
?
47. How to Fix This?
How to Fix This?
Copyright Petro Porchuk 2010. Lohika
?
Total Quality Management (TQM)
Test and Development Processes Improvement
48. (7)
(7) Assessing and Improving the
Assessing and Improving the
Development and Test Processes
Development and Test Processes
Copyright Petro Porchuk 2010. Lohika
General Techniques and Approaches
Total Quality Management
Kaizen
Six Sigma
Improving the Software Development Processes
Capability Maturity Model Integration (CMMI)
SPICE
Evaluating the Test Processes
Testing Maturity Model (TMM)
Test Process Improvement (TPI)
49. 7.1 Total Quality Management (TQM)
7.1 Total Quality Management (TQM)
Copyright Petro Porchuk 2010. Lohika
QUALITY is in the center. No instructions – it's an attitude.
http://www.edrawsoft.com/TQM-Diagrams.php
50. 7.2 Kaizen
7.2 Kaizen
(Japanese for
(Japanese for Kai "
Kai "improvement" or
improvement" or Zen
Zen "change for the better")
"change for the better")
Copyright Petro Porchuk 2010. Lohika
Intensive use of the employee suggestion system
Appreciation and regard for employees' striving for improvement
Established small group discussion circles addressing defects and improvement
suggestions
”Just in time” production to eliminate wasted effort
5 Ss process for the improvement of workspaces:
Seiri – tidiness; old and useless items have no place at the workspace
Seiton – orderliness; ”everything” has its place for quick retrieval and storage
Seiso – cleanliness; keep the workspace clean and tidy at all times
Seiketsu – standartization of all practices and processes
Shitsuke – discipline; all activities are performed in a discipline way
”Total Productive Maintenance” for maintenance and servicing of all means of
production
BUT: difficult to apply in european culture such strong Jananese rules
51. 7.3 Six Sigma
7.3 Six Sigma
(by Motorola, USA in 1981)
(by Motorola, USA in 1981)
Copyright Petro Porchuk 2010. Lohika
Six Sigma seeks to improve the quality of process outputs
by identifying and removing the causes of defects (errors)
and minimizing variability in manufacturing and business
processes.
The term six sigma originated from terminology associated
with manufacturing, specifically terms associated with
statistical modelling of manufacturing processes.
The maturity of a manufacturing process can be described
by a sigma rating indicating its yield, or the percentage of
defect-free products it creates.
A six-sigma process is one in which 99.99966% of the
products manufactured are free of defects (), compared to a
one-sigma process in which only 31% are free of defects.
BUT: Requires a lot of statistics from the Test Manager
http://en.wikipedia.org/wiki/Six_Sigma
52. 7.4 CMMI
7.4 CMMI
(Software Engineering Institute (SEI, 2002))
(Software Engineering Institute (SEI, 2002))
Copyright Petro Porchuk 2010. Lohika
http://en.wikipedia.org/wiki/Capability_Maturity_Model_Integration
53. 7.5 SPICE
7.5 SPICE ("Software Process Improvement and Capability Evaluation")
("Software Process Improvement and Capability Evaluation")
(ISO/IEC 15504 standard)
(ISO/IEC 15504 standard)
Copyright Petro Porchuk 2010. Lohika
http://www.kuglermaagusa.com/tools_spice15504_com.html
0 ”Incomplete process” | General Failure to perform the bace practices in the Process
54. 7.6 Testing Maturity Model (TMM)
7.6 Testing Maturity Model (TMM)
(1996 Illinois Institute of Technology, Chicago)
(1996 Illinois Institute of Technology, Chicago)
Copyright Petro Porchuk 2010. Lohika
Based on CMM:
Levels Process Areas
1 – Initial No process identified
2 – Phase Definition Test Policy and Goals
Test Planning
Test Techniques and Methods
Test Environment
3 - Integration Test Organization
Test Training Program
Test Life Cycle and Integration
Control and Monitor
4 – Management and Measurement Peer Reviews
Test Measurement
Software Quality Evaluation
5 – Optimization Defect Prevention
Quality Control
Test Process Optimization
55. 7.7 Test Process Improvement (TPI)
7.7 Test Process Improvement (TPI)
(Sogeti, Netherlands. http://www.sogeti.com)
(Sogeti, Netherlands. http://www.sogeti.com)
Copyright Petro Porchuk 2010. Lohika
Key / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
1 Test strategy A B C D
2 Life-cycle model A B
3 Moment of involvement A B C D
4 Estimating and planning A B
5 Test Specification techniques A B
6 Static test techniques A B
7 Metrics A B C D
8 Test tools A B C
9 Test environment A B C
10 Office environment A
11 Commitment and motivation A B C
12 Test functions and training A B C
13 Scope of methodology A B C
14 Communication A B C
15 Reporting A B C D
16 Defect management A B C
17 Testware management A B C D
18 Test process management A B C
19 Evaluation A B
20 Low-level testing A B C
56. 7.8 TPI Assessment
7.8 TPI Assessment
Copyright Petro Porchuk 2010. Lohika
Key / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
1 Test strategy A B C D
2 Life-cycle model A B
3 Moment of involvement A B C D
4 Estimating and planning A B
5 Test Specification techniques A B
6 Static test techniques A B
...
Key / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
1 Test strategy A B C D
2 Life-cycle model A B
3 Moment of involvement A B C D
4 Estimating and planning A B
5 Test Specification techniques A B
6 Static test techniques A B
...
Current..
Required..
57. (7) Summary
Copyright Petro Porchuk 2010. Lohika
✔ It is up to the individual company to decide how well a
particular model is suited toi it's own specific
circumstances.
58. CTAL TM – Part 2
Part 1
1. Introduction
2. Test Process and Test Tools
3. Testing in the software Life Cycle
4. Test Policy and Test Handbook
5. The Test Plan
6. Test Control
7. Assessing and Improving the Development and Test Processes
Part 2
8. Deviation Management
9. Risk Management and Risk-Oriented Testing
10. Staff Qualification and Skills
11. Test Metrics
12. Selecting and Implementing Test Tools
13. Standards Copyright Petro Porchuk 2010. Lohika
62. What is Risk?
What is Risk?
Copyright Petro Porchuk 2010. Lohika
?
IT Project = Risk = Probability * Damage
63. (9)
(9) Risk Management and Risk-Oriented Testing
Risk Management and Risk-Oriented Testing
Copyright Petro Porchuk 2010. Lohika
Risk Management
Identification of the risk context
Risk identification
Risk analysis and risk evaluation
Risk control
Risk verification and monitoring
Risk-Oriented Test Plan Creation and Test Prioritization
Failure Modes and Effect Analysis (FMEA)
64. 9.1.1 Identification of the Risk Context
9.1.1 Identification of the Risk Context
Copyright Petro Porchuk 2010. Lohika
65. 9.1.2 Risk Identification. Risk Categories
9.1.2 Risk Identification. Risk Categories
Copyright Petro Porchuk 2010. Lohika
Risk Category Influence Prediction Example
External risk
Cannot be influenced Difficult to predict •Lightning Damage
•Out of Power
Strategic risk
Difficult to influence Easy to predict •Changing a Corporate
standard etc
Project risk
Easy to influence Easy to predict •Sales Problems
•Technical Risks
•Servers unavailability
•Defect Correction
Risks
Product risk
Can be influenced Easy to predict •Customer suffering
from product failures
•Poor Quality affects
reputation
66. 9.1.3 Risk Identification Techniques
9.1.3 Risk Identification Techniques
Copyright Petro Porchuk 2010. Lohika
Expert Interviews and Questionnaires
Independent Estimations
Risk Workshops
Risk Brainstorming (Worst Case Scenarios)
Use of risk templates and checklists
Experiences from completed projects
67. 9.1.4 Risk Evaluation
9.1.4 Risk Evaluation
Copyright Petro Porchuk 2010. Lohika
F – the frequency of execution of use (a function)
C – criticality. Represents the worst possible effects of a function not working well (at all)
Rp – The Project risks
Rt – Technical Product risks
Rb – The Commercial Product risks
Risk Factor:
RF=
RtRbRp
3
F∗RbC∗Rt
68. 9.2 Risk Oriented Test Plan Creation
9.2 Risk Oriented Test Plan Creation
Copyright Petro Porchuk 2010. Lohika
Target-oriented test plan creation:
applying appropriately different test techniques and test
intensities to cover system functions with different risks
Prioritized testing:
giving areas with higher risks higher priority and testing them
earlier
Residual risk awareness:
identifying the residual risks that remain in the delivered
software and are due to reductions in test or nonperformance of
planned tests
69. 9.3.1 Risk Based Testing.
9.3.1 Risk Based Testing.
Failure Modes and Effect Analysis (FMEA)
Failure Modes and Effect Analysis (FMEA)
Copyright Petro Porchuk 2010. Lohika
FMEA is a Method used to identify potential error
types and their effect on a system.
FMEA Procedure:
1. Generate a list of all the risk factors.
2. Specify the type and probability of falures for each factor.
3. Investigate the effect of failures on other factors, applying, e.g., simulations.
4. Investigate the effect on Project Planning.
5. Identify possibilities for detecting the failure.
6. Identify possibilities for compensating the failure.
7. Identify possibilities for preventing the failure.
8. Define measures to avoid defects.
9. Evaluate the effect of the proposed measures.
10. Document the results.
70. 9.3.2 Risk Priority Number.
9.3.2 Risk Priority Number.
Failure Modes and Effect Analysis (FMEA)
Failure Modes and Effect Analysis (FMEA)
Copyright Petro Porchuk 2010. Lohika
FMEA defined the Risk Priority Number (RPN).
O – Probability of Occurrence
E – Expected Damage
D – Probability of Detection
RPN =O∗E∗D
71. (9)
(9) Summary
Summary
Copyright Petro Porchuk 2010. Lohika
✔ Risk = Rrobability * Damage
✔ Testing is a preventive measure, reducing risks
✔ Based on the most important risks, suitable test
techniques are to be selected, and tests are to
prioritized to allow tests covering high risks to be
executed early and with the appropriate effort
✔ Prioritization and risk estimation criteria, defined in the
Test Plan, can be applied to test objects and test
cases
72. Skilled people as the key success factor
Skilled people as the key success factor
Copyright Petro Porchuk 2010. Lohika
TOYAKO, JAPAN - JULY 08: (L-R) German Chancellor Angela Merkel, U.S. President George W. Bush, Japanese Prime Minister
Yasuo Fukuda, French President Nicolas Sarkozy and Russian President Dmitry Medvedev plant the memorial tree after a working
session at the Windsor Hotel Toya on July 8, 2008 in Toyako, Hokkaido, Japan.
http://www.life.com/image/81853290
73. (10)
(10) Staff Qualification and Skills
Staff Qualification and Skills
Copyright Petro Porchuk 2010. Lohika
Individual Skills
Functional Team Roles
Social Team Roles
The Communication Factor
The Motivation Factor
74. 10.1 Individual Skills
10.1 Individual Skills
Copyright Petro Porchuk 2010. Lohika
The ”perfect” test team member has exellent technical knowledge and high social competency.
There is NO PERFECT Individual from the scratch. Need to develop.
IT Expert as QA:
Great technical competency
DANGER: In love with technology → loses focus
Application Expert as QA:
Great domain expert. Knows non-func. requirements
DANGER: Blinkered atitude → doesn't seek for improvements
Social Competence
The ability to familiarize quickly with complex domains
The ability to detect defects. (”professional ressimism”)
The ability to cope with criticism adequately. Diplomacy
Courage to leave some gaps
Discipline, exactitude, patience, tolerance
The ability to work in a team and to communicate
75. 10.2 Functional Team Roles
10.2 Functional Team Roles
Copyright Petro Porchuk 2010. Lohika
Test Manager
Leads the Test Team
Test Planning and Control, Test Reporting
Human Resource Management and Test Process Improvement
Test Designer (QA Analyst)
Creation and Maintenance of Test Specification. Prioritizing
Defining and Apply Test Design Techniques
Test Automator
The Test Team's Programmer
Developing Automation Tests, Test frameworks and Tools
Test Administrator
Installation, Administration and Maintenance of the Test Environment
Installing and set up of the System Software (Operating Systems, Databases, Application Servers)
Tester
Execution the tests and documenting the test results
Accuracy and Creativity
Expert
Supporting L/P testing, database or network testing etc
76. 10.3 Social Team Roles
10.3 Social Team Roles
Copyright Petro Porchuk 2010. Lohika
Type Descriptors Strengths Weakness
Monitor Evaluator Strategic Power of judgment Lacks drive and the ability to
inspire others
Shaper Dynamic, open-minded,
result oriented
Lot's of drive, fight
inefficiency, experts pressure
Prone to irritate and to hurt
others
Planter Individualistic Creative. Lot's of intellectual
power
Ignores practical details and
instructions
Completer Straight, anxiuos Ability to complete things,
perfectionism
Inclined to worry undully,
reluctant to delegate
Team Worker Cooperative, mild, sensitive Ability to cope with different
situation and people,
promotes team spirit
Can be easily influenced
Implementer Conservative, dutiful Hard-working, disciplined Inflexible. Rejects unproved
ideas
Co-ordinator Self-confident, mature Recognizes individual
talents. Good at clarifying
goals
Has limited creativity
Resource Investigator Extrovert, enthusiastic,
communicative
Likes to develop contacts,
picks up new ideas, reacts to
challenges
Too optimistic, tend to lose
interest after initial
enthusiasm
Specialist Single-minded, self-starting,
dedicated
Provides knowledge and
skills in rare supply
Contributes only a narrow
front, overlooks the ”big
picture”
77. 10.4 The Communication Factor
10.4 The Communication Factor
Copyright Petro Porchuk 2010. Lohika
Communicate problems accurately
Test Team External Communication
Tester → Developer
Developer → Tester
Test Manager → Projects Manager
Project Manager → Test Manager
Tester → User
Test Team Internal Communication
Regular Team Meetings
Knowledge Transition
78. 10.5.1 The Motivation Factor
10.5.1 The Motivation Factor
Copyright Petro Porchuk 2010. Lohika
http://media.photobucket.com/image/funny workers/geet_kunal/happy-best-wishes-monday/rg272945.jpg
79. 10.5.2 The Motivation Checklist
10.5.2 The Motivation Checklist
Copyright Petro Porchuk 2010. Lohika
The schedule is a motivating factor
The resources provided are a motivating factor
The test process is a motivating factor
The task is a motivating factor
The team will feel valued, recognised and
motivated
http://www.imbus.de/fileadmin/imbus_repository/Downloads/imbus_Allgemein/Dokumente/Checkliste-Motivation_E.zip
80. (10)
(10) Summary
Summary
Copyright Petro Porchuk 2010. Lohika
✔ A member of the test team should be IT professional, have
knowledge from the application domain and have social competence
✔ The different tasks in the test process are reflected in the different
roles within the test team
✔ Becides professional role, each team member should play a social
role in the team
✔ Detected problems should be communicated to the external parties
with relevan communication style
✔ Test Manager should organize regular internal meetings
✔ The Test Manager should provide a healthy motivation environment
and professional working condition for the team. Part of this task is to
adequately represent the role and contribution of the test team within
the organization
81. Why do the People change job?
Why do the People change job?
Copyright Petro Porchuk 2010. Lohika
Stupid Manager
Far from Home
Low Salary
Not Allowed to use Facebook
82. (11)
(11) Test Metrics
Test Metrics
Copyright Petro Porchuk 2010. Lohika
Why do we need Metrics?
Presenting Measurements
Test Metrics Types
Residual Defect Estimations and Reliability
83. 11.1 Why do we need Metrics?
11.1 Why do we need Metrics?
Copyright Petro Porchuk 2010. Lohika
The Test Metrics is the Tool for Control
Tom DeMarco on Measurement:
„You can not control what you can not measure.
Measurement is the prerequisite to management
control.“
from Tom DeMarco
American Consultant, 1982
84. 11.2 Presenting Measurements
11.2 Presenting Measurements
Copyright Petro Porchuk 2010. Lohika
Diagrams help you to visualize measurement values and their progression
85. 11.3.1 Test Metrics Types
11.3.1 Test Metrics Types
Copyright Petro Porchuk 2010. Lohika
Test Case Based Metrics
Number of Planned, Specified Test Cases
Number of Created Test Procedures
Number of Changed, Deleted Test Cases
Number of Run, Passed, Failed, Blocked TC
Test Object Based Metrics
Number of tested functions / Total number of functions
Number of executed TC / Number od specified TC epr function
Percentage of all requirements covered by Test Cases
Number of platforms covered by test
Percentage of interfaces tested
Number of lines of codes or executable instructions (kilo lines of code – KLOC)
Number of the covered paths in the control flows graph
Percentage of branches in the control flow graph that were covered
86. 11.3.2 Test Metrics Types
11.3.2 Test Metrics Types
Copyright Petro Porchuk 2010. Lohika
Defect - Baced Metrics
Bug Detection Percentage
Number of defects relative to criticality (defect severity)
Number of defects per status (defect correction progress)
Defect report status
Cost and Effort-Based Metrics
Number of person-days for test planning or specification
Number of person-days for the creation of test procedures
Evaluating Test Effectiveness
Degree of Defect Detection
Confidence Level
Test Effectiveness
DDD=
numberof weighted defects during test
total numberof defects found
CL=1−
numberof weighted defects
number of executed test cases
∗degree of test coverage
TE=DDD∗CL
87. 11.4 Residual Defect Estimations and Reliability
11.4 Residual Defect Estimations and Reliability
Copyright Petro Porchuk 2010. Lohika
Three options to evaluate Test Process based on defects found
Estimate number of defects inherent in the system prior to test and estimate the number of
defects to be found
Monitor the Defect Detection Percentage
Inject bugs into the program code and monitor the proportion to be found during testing
Reliability Growth Model
Target numberof Defects=
residual numberof defects
effectiveness percentage
88. (11)
(11) Summary
Summary
Copyright Petro Porchuk 2010. Lohika
✔ For each Test Level suitable Test Metric should be
applied
✔ Ensure compliance, if necessary through audits
89. (12)
(12) Standards
Standards
Copyright Petro Porchuk 2010. Lohika
General Understanding of Standards
Standards related to Software Testing
Standards in the phases of the Test Process
90. 12.1 General Understanding
12.1 General Understanding
Copyright Petro Porchuk 2010. Lohika
Standards are developed and maintained by national and
international organizations
ISO – International Organization od Standardization
ANSI – American National Standard Institute
BSI – British Standard Institution
IEEE – Institute of Electrical and Electronics Engineers
Standard Types
Corporate Standards
Best Practices and Technical specifications
Domain Specific
Generally Applicable Standards
91. 12.2 Main Standards related to Software Testing
12.2 Main Standards related to Software Testing
Copyright Petro Porchuk 2010. Lohika
Standard Description
BS 7925-1 British Standard: Software Testing, Part 1: Vocabulary, 1998
BS 7925-2 British Standard: Software Testing, Part 1: Software Component Testing, 1998
ISO 9000 A family of standards for quality management systems
IEEE 610.12 1990 IEEE Standard Glossary of Software Engineering Terminology
ISO 90003 Software Engineering Quality Management Standard
IEEE 730 IEEE Standard for Software Quality Assurance Plans, 2002
IEEE 1008 ANSI/IEEE 1008-1987 IEEE Standard for Software Unit Testing
IEEE 1012 IEEE Std 1012-1998 IEEE Standard for Software Verification and Validation
ISO 9126-x IEEE Std 1044-1993 IEEE Standard Classification for Software Anomalies
ISO 9241 ISO 9241-11:1998. Ergonomic requirements for office work with visual display terminals
IEEE 1044 IEEE Std 1044-1993 IEEE Standard Classification for Software Anomalies
IEEE 829 IEEE 829-1998, Standard for Software Test Documentation
IEEE 1059 IEEE Std 1059-1993 IEEE Guide for Software Verification and Validation Plans
92. 12.3 Standards in the phases of the Test Process
12.3 Standards in the phases of the Test Process
Copyright Petro Porchuk 2010. Lohika
Type of
Standard
Test Phase
Terminology
and contracts
Processes Products Documentation Methods and
techniques
Test Planning
Test Control
BS 7925-1
ISO 2382
ISO 9000
ISO 12207-0
IEEE 610.12
ISO 12207-0
ISO 9000-3
IEEE 730
IEEE 1008
IEEE 1012
IEEE 1061
ISO 9126-x
ISO 12119
ISO 12207-1
ISO 15026
IEEE 982.1
IEEE 1228
IEEE 730
IEEE 829
IEEE 1012
IEEE 1228
ISO 12207-2
ISO 16085
IEEE 730.1
IEEE 982.2
IEEE 1059
Test analysis
Test design
ISO 14598-4
ISO 15408
IEEE 1062
IEEE 1228
IEEE 1008
IEEE 1012
ISO 15939
ISO 9241
ISO 12119
IEEE 1044
IEEE 982.2
IEEE 829
IEEE 1063
BS 7925-2
IEEE 1028
ISO 60812
EN 61508-3
EN 50128
IEC 61025
ISO 5806
Test realization
Test execution
IEEE 1008 ISO 9241
IEEE 828
IEEE 1209
IEEE 1348
IEEE 829
IEEE 1209
IEEE 1348
IEEE 1028
ETSI 201-873
IEEE 1042
IEEE 1209
IEEE 1348
Test evaluation
Reporting
IEEE 1008 IEEE 982.2
IEEE 1044
IEEE 829 IEEE 1044.1
ISO 12119
93. (13)
(13) Summary
Summary
Copyright Petro Porchuk 2010. Lohika
✔ Test Manager shouldn't reinvent the wheel, use
standards for Testing
✔ Ensure compliance, if necessary through audits