Test Process Improvement with TPI NEXT - what the model does not tell you but you should know
1. Lvov, Ukraine
Nov 7, 2013
Test Process Improvement
with TPI Next
What the model
does NOT tell you
BUT you should know
Tapani Aaltio
Sogeti Finland Oy
2. Using a Maturity Model as a Reference Helps in
Structuring and Communicating the Improvement Actions
CMM(i)
TMM(i)
…How can you tell if you do the right things (right)…
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
3. Improving the Test Process is a
Continuous Change Towards Better Maturity
Create
Awareness
Determine
Goal, Scope
and Approach
Assess Current
Situation
Define
Improvements
Evaluate and
Redirect
Make a Plan
of Action
Implement
Actions
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
4. The TPI NEXT Model Consists of 16 Key Areas,
Four Maturity Levels and 157 Checkpoints
Test maturity matrix
Key areas
Maturity levels
Checkpoints
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
5. In the TPI NEXT Model, the Field of Testing
is Divided into 16 Key Areas in Three Groups
1 Stakeholder commitment
2 Degree of involvement
Stakeholder
Relations
3 Test Strategy
4 Test Organization
5 Communication
6 Reporting
7 Test Process Management
Test Management
8 Estimating and Planning
9 Metrics
10 Defect Management
11 Testware Management
12 Methodology Practice
13 Tester Professionalism
Test Profession
14 Test Case Design
15 Test Tools
16 Test Tapani Aaltio: What the TPI Model does not tell you – but you should know
Environments
SQA Days Lvov, Nov 7, 2013
6. The TPI Model has Four Maturity Levels, with
Requirements for Each Key Area on Each Maturity Level
• Adhoc Activities
Initial
• Doing the right
Things
Controlled
• Doing it the
right way
Efficient
• Continuously
adapting
Optimizing
Description of Key Area ”Test Strategy” on
different maturity levels
The test strategy enables the distribution of the
test efforts and resources among the test levels
and test activities.
The test strategy is used to achieve adequate
balance between analyzed product risks, test
coverage and available efforts and
resources, involving all stakeholders.
The method of the test strategy is maintained well
to ensure easy and valid application.
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
7. The TPI Model has Four Maturity Levels, with 2-4
Checkpoints for Each Key Area on Each Maturity Level
• Adhoc Activities
Initial
• Doing the right
Things
Controlled
• Doing it the
right way
Efficient
• Continuously
adapting
Optimizing
Examples of check points of Key Area ”Test
Strategy” on different maturity levels
The principal stakeholder agrees with the
documented test strategy.
The test strategy is based on product risk analysis.
All relevant stakeholders agree with the
documented test strategy.
The overlap and gaps in test coverage between test
levels or test types are well considered.
The process of creating the test strategy is
evaluated regularly.
The test strategy is evaluated against metrics on
indicents that occur in production.
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
8. The Results of an Assessment are
Presented Visually in a Testing Maturity Matrix
16 Key Areas
• Doing the right
Things
• Doing it the
right way
• Continuously
adapting
Controlled
Efficient
Optimizing
Total 157 Check Points
63% of
checkpoints
met
12% of
checkpoints
met
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
9. To Understand the Whole and the Details,
Use Multiple Channels to Gather Information
Interview the right
people with different
points of view
Investigate documents
to find evidence
“Hang around
in the lobby”
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
10. Defining the Improvements is a Crucial Step –
And it is Fully Supported by the TPI NEXT Model
Create
Awareness
Determine
Goal, Scope
and Approach
Assess Current
Situation
Define
Improvements
Evaluate and
Redirect
Make a Plan
of Action
Implement
Actions
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
11. The TPI Model Offers Concrete Improvement Suggestions
for Each Key Area on Each Maturity Level
• Adhoc Activities
Initial
• Doing the right
Things
Controlled
• Doing it the
right way
Efficient
• Continuously
adapting
Improvement Suggestions of Key Area ”Test
Strategy” on different maturity levels
Include various stakeholders in
Product Risk Analysis and Test Strategy.
Investigate possible overlaps between different test levels.
Consider combining test levels, if possible.
Define the role of Test and Evaluation Coordinator,
to co-ordinate and monitor all test and evaluation activities.
Optimizing
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
12. The Clusters (A, B, C, … ) Help in Finding the Balanced
Improvement Paths to a Higher Maturity Level
Prioritize Key Areas to
16 Key Areas
create Business Driven• Doing the right
Improvement Paths Things
Controlled
• Doing it the
right way
• Continuously
adapting
Efficient
Optimizing
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
13. The Complete TPI® NEXT Model
Test Maturity Matrix
Key areas
Clusters
Maturity levels
Checkpoints
Improvement
suggestions
Enablers
e.g. Requirements
Management and
Project management
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
14. Making a Realistic Plan of Action Requires Detailed
Planning – Don’t be Greedy and Try to Do Too Much!
Create
Awareness
Determine
Goal, Scope
and Approach
Assess Current
Situation
Define
Improvements
Evaluate and
Redirect
Make a Plan
of Action
Implement
Actions
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
15. To Sell your Ideas to Management,
Remember Use Both Facts and Feelings
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
16. If the Planned Improvement Actions cannot be
Implemented, the Whole Investment has Gone to Waste
Create
Awareness
The fun part
Determine
Goal, Scope
and Approach
The tough part
Assess Current
Situation
Define
Improvements
Evaluate and
Redirect
Make a Plan
of Action
Implement
Actions
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
17. Typical Impediments for Improvement –
as Seen by Customers
Resource availability for the Improvement Project
Resource competencies in testing and quality assurance
Cultural change – not easy to implement
Improving feels like “an additional effort”
No clear benefits seen
Unclear scope and focus
No owner for the improvement
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
18. The Organization must be Ready to Accept Declining
Productivity in the Early Phases of Implementation
Productivity
Work systematically to
reach a higher level
Goal
Current
level
Don’t give
up here!
Time
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
19. In a Successful Improvement Project, Assessing the
Current Situation Represents only 10-30% of Total Work
Example Improvement Project Work Estimates
# Task
Sogeti
(days)
Customer
(days)
% of total
1 Assessing the Current Situation
20
5
12,5%
2 Organizing and training the
Customer Test Improvement Team
20
20
20,0%
3 Define General Test Agreements
and Test Processes
40
10
25,0%
4 Support for Implementation of the
new process in projects
60
25
42,5%
Total
140
60
100,0%
19
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
20. TPI NEXT Provides Tangible Profit to Business
Improved quality (60%)
Prevented damage by finding a majority of critical & major defects in
testing phase
Shorten time for testing (24%)
Reduction in test duration due to a uniform controlled test strategy and
risk based testing approach
Reduce cost (35%)
Less test cases & higher coverage = less test effort
Shorter fix cycle
Deadline?
Implement
Test
Fix
Test
Fix
Test
Etc
Why not do it right the first time – and minimize the fixing cycle?
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
21. Quality Gates Help in Catching Defects in the Same
Phase Where They are Injected – No Faults Forward!
Project without quality gates
167
Defects per Test Level
Waste,
customer
dissatisfaction
109
74
54
47
34
19
21
ST
Project with quality gates
6
5
FAT
SIT
E2E
29
UAT
Aftercare
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
22. ROI calculation for No Faults Forward! without Quality Gates vs. with Quality Gates
Baseline # defects (without quality gates)
Pilot # defects (with quality gates)
276
63
Reduction # defects
213
Test cost per defect
≈€ 1.500,00
Rework cost per defect
≈€ 2.000,00
Total cost per defect
≈€ 3.500,00
# Defects * cost per defect
Investment QA-consultant
Cost reduction
€ 745.500,00
€ 30.400,00
€ 715.100,00
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
23. Summary
TPI NEXT is an easy-to-use, practice based assessment model for your test
process
The model provides balanced, business-driven improvement paths to reach
a higher level of maturity
The implementation of the improvements is hard work. The following ideas
might help you achieve good results
Engage the organisation from day one
Use facts and feelings to sell your improvements to key people and management
Find an executive sponsor – and keep him/her engaged
Proceed in small steps
Demonstrate added value early
Don’t give up before you have reached a higher level of productivity
For more information on TPI NEXT, navigate to
http://www.tmap.net/en/tpi-next
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
24. Information on TPI NEXT
is Easily Available in Different Formats
Download
Android App
at Google Play
Check out online resources at
http://www.tmap.net/en/tpi-next,
Buy the book at
including the Excel-based Test Maturity Matrix Tool
http://www.utn.nl/
and read it!
Pass the EXIN
TPI Foundation
Certificate exam
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
25. Now it’s your Turn – Questions, Answers and Discussion...
Thank you for your attention, this was
“Test Process Improvement with TPI Next –
What the model
Does NOT tell you
BUT you should know”
Tapani Aaltio
Sogeti Finland
tapani.aaltio@sogeti.com
Lvov, Ukraine
Nov 7, 2013
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
27. TPI Next Can be Applied together with CMMi,
Using CMMi-driven Clusters
Download white paper on TPI Next and CMMi at http://www.tmap.net/en/tpi-next/publications
Tapani Aaltio: What the TPI Model does not tell you – but you should know
SQA Days Lvov, Nov 7, 2013
28. The Structured Test Process According to TMap®
Specification
Preparation
Creating test
specifications, Defining central
starting points, Specifying test
object intake
Collection of the test basis,
Testability assessment
Execution
Control
Management,
Monitoring,
Reporting,
Adjusting
Ctrl
Plan
Prep
Spec
Exec
Intake of test object,
Preparing the starting points,
Executing the (re)tests,
Checking and assessing the
test results
Comp
Completion
Infra
Planning (main tasks)
Analysing the product risk,
Determining the test strategy,
Estimating the effort,
Defining test products,
organisation and infrastructure
Evaluating the test
process, Preserving testware
Infrastructure management
Specifying the infrastructure,
Realising the infrastructure,
Specifying the infrastructure intake,
Intake of the infrastructure,
Maintaining the infrastructure,
Preserving the infrastructure the TPI Model does not tell you – but you should know
Tapani Aaltio: What
SQA Days Lvov, Nov 7, 2013
Notas do Editor
Many companies lose money or even their reputation because of bad quality IT-systems. When quality is bad, it sounds logical to start improving testing practices. The TPI NEXT –model is a great tool for assessing where you are and for creating a coherent roadmap for improvement.However, the improvement actions fail too often. Sometimes the plans are not realistic, the right stakeholders are not involved, or management doesn’t really support the initiatives. These – and other - potential impediments should be taken into account already from day one in the TPI assessment. In this talk, I will shortly present the TPI NEXT -model and discuss my practical experiences on how to succeed in improving - not only the test process, but also the whole software development life cycle. The topics discussed include both success stories and LFMF’s (=learn from my failure). In short, I will tell you the things you should know, but can't find in the TPI NEXT book.1. Short presentation of the structure and main points of the TPI NEXT model, including its connections to other processes in the SDLC2. Assessment of the strengths and weaknesses of the TPI NEXT Model, based on several assessments with different customers3. Tips on how to conduct a TPI assessment in such a way that when the assessment is done, everything is set up for a fruitful improvement project.
Create awarenessMostly awareness is there, because of quality problems in prod, long test & fix cycles, TODODetermine goal, scope and approachThe scope is normally a project or two, it’s important to have a limited scopeTwo different packages: full TPI or Quick ScanBe sure to agreewith the stakeholders – and understand -what the purpose of the assessment is - or is it just a nice to knowmeasurementAssess current situationFirst thing to do: choose and engage the right people to work with you – these will be your change agents later!
Degree ofinvolvement: howtightlytesting is coupledwith the wholeSwdevelopmentlifecycle
TPI is not mathematics – it is not b&wTODO example of ”notmathematics”
DefineImprovementsFocused improvement suggestions and stepwise improvement (in short cycles) are two strengths of the model!Unfortunately, itdoesnottellwhatyourimprovementvelocitycouldbe - howmanycheckpoints / monthDon’toverdoit, evenRomewasn’tbuilt in a dayShow the customeryour vision of the targetsituationOn the other hand, you sometimes need “revolution and evolution”!Not all areas are equally important? If you have the wrong test cases, you are lost! BUT to define the right cases, you need other skills, e.g. business knowledge
Make a plan of actionthe book covers this on only 2 pages!Considerbothquickwins and longertermgoalsH2 sellyourideas to the organization - Rememberthatdecisionsare made based on bothfacts and feelings.
Youcannotsell the best test process to everybody with a detaileddescription of the process itself!Create a vision of an easy life of somethingnice.
Evaluate and redirectThis belongs to the fun part!this makes continuous improvement possible, but is covered only on 1 page!Implement actionsthis is the hardest part, but the book only covers this on 1 page!people should have an understanding of the whole, “the big picture” to understand the meaning of “improvements” – training neededcreating the awareness, training, coaching, support neededCommunication using all possible channels neededthis cannot be accomplished without a permanent test / quality role in place to monitor progress of the implementationThis role should not do it him/herself, but motivate and make sure somebody in the project doesButevenwithallthis in place, youaregoing to meetchangeresistance…
Rememberyourultimategoal: clientsatisfaction - but at whatpoint in time?
Revolution and evolution!Sohowdogetupthatcurve?
Whereas, the support forimplementationcouldeasilybealmosthalf of the effort!Peoplefeelthattheyhave to domore…. … and ifyoudoallthis, whatdoyouget? – is itpossible to a systemwhich is good, cheap and fast at the sametime?
Whynotfix the defects in the phasewheretheyareinjected?
Transition:… and ifwecalculate the savings in money…
CreatedbySogeti, butavailable for everybody in differentformats.No licensefees, just by the bookordownload the application!