2. Purpose
• This presentation outlines a number of key principles that could be
applicable for any test organization
• By applying these key principles to a test organization it becomes
more efficient and effective
• Not everything is applicable in every context, but the key principles
should be generic in nature and usable under most circumstances
3. Testing as a
Service
Right Smart
Competence Automation
Empowering
Useful Metrics
Tools
Key Principles
Context
Risk-based
Dependent
Testing
Test Methods
Customer
Changing
Focused
Requirements
Reporting
4. Key Principle: Testing as a Service
• A tester is not a quality police – clients make decisions based on a
service the tester provides [1]
• A test organization should provide an understandable and simple
service catalogue of how they can help their clients get the
information they need to make the correct decisions
• A test organization should be able to support their clients in selecting
what services are most suitable for their specific needs
• A test organization should always be able to show and explain their
value
5. Key Principle: Smart Automation
• Smart test automation is not automating a set of manual tests [2]
• Smart automation is designing a good automated test scope from the
beginning
• Manual tests are not designed to be automated – create automated
tests that are suitable for automation, don’t force automation on
something that was designed to be manual
• Always have a return-on-investment analysis, and consider
maintenance, portability and re-usability costs
• Drive testability in the code so that automation becomes a natural
part of the development process
6. Key Principle: Useful Metrics
• Not everything that counts can be measured. Not everything that can
be measured counts. [3]
• Metrics should be used to pinpoint where an analysis or audit must
be conducted, not as hard evidence
• Always consider what kind of behavior a certain metric will promote
before communicating it
• Never be afraid to measure, but understand that metrics never give a
complete picture
7. Key Principle: Risk-based Testing
• All testing should be based on a proper risk analysis [4]
• Clients decide what they want, but a tester must be able to explain
why a certain scope is more or less effective for the clients needs and
acceptable risk levels
• Use as much empirical data as possible for these risk analysis, don’t
just go on a gut feeling – quantify that feeling when possible
• Inform the client of the costs of different levels of uncertainty and
risk; if the client wants everything tested and no risk it will cost an
indefinite amount, if the clients accepts any risk, then nothing is
tested, and the cost is zero
8. Key Principle: Customer Focused Reporting
• Don’t ask the client what information he wants – ask what decisions
the clients intend to take based on the information
• Tailor the test and defects reports based on the clients needs;
qualitative and quantitative information can be equally powerful –
but always back up statements with facts
• A tester must communicate what confidence the client can have in
the results – just because no issues are found does not mean that the
software is bug free
• But always remember; “We don’t manage the project managers. We
don’t decide what information they have to give to the people they
report to.” [5]
9. Key Principle: Changing Requirements
• Don’t expect complete requirements up front
• When parameters change, the tester must adapt to meet new
stakeholder needs; be adaptable to changing requirements during a
project
• If requirements are needed and none are available, write down
expected behaviour and request stakeholder acknowledgement –
communication with stakeholders is key when requirements are
unclear
• Try to minimize unnecessary test artefacts – combining
requirements and test cases into one entity can be one way forward
10. Key Principle: Context Dependent Test
Methods & Approaches
• The most effective and efficient testing is context dependant [6]
• It is the tester’s job to find the most effective and efficient method for
each context
– Scripted testing - Exploratory testing
– Black box - Gray box - White box
– Manual - Automated
– Tool supported or not
– Boundary analysis -Equivalence partitioning -Fuzz testing - State transition testing
– Etc.
11. Key Principle: Empowering Tools
• A tester should spend a majority of the time testing – streamline
everything else with the help of tools [7]
• Risk-analysis tools help the tester perform more efficient and
effective risk analysis
• Integrated reporting tools help the tester file defects without
changing systems [7]
• Tools to support time efficient test plans [8]
• Record and playback framework for repeating tests [9]
12. Key Principle: Right Competence
• There are many different tester roles that each require a diverse set
of skills – one size does not fit all
• Understand what roles are needed in a specific organization and
secure that the testers have the skills needed to fill those roles
• Make sure that every tester has an idea of what their career might
look like in five years, based on their specific skillsets [10]
• Understand that it is not a numbers game – do not hire more testers
to solve a problem – secure that you have the right testers [7]
• More does not equal better – it can sometimes lead to more entropy
and chaos
13. Summary
• These key principles should be applicable for most test organizations,
even if there will always be some exceptions
• While they do not cover everything, they should cover the most
critical areas that most test organizations need to think about
• There is no best practice – but this is a good start to help a test
organization finding a best practice for their context
14. Reference
[1] Ten Things about Testing that should die: Egocentricity
http://scott-barber.blogspot.com/2011/11/10-things-about-testing-that-should-die.html
[2] Exploring Test Automation
http://angryweasel.com/blog/?p=412
[3] Albert Einstein qoute
http://www.famous-quotes.net/Quote.aspx?Not_everything_that_counts_can_be_measured
[4] Risk-based Testing
http://en.wikipedia.org/wiki/Risk-based_testing
[5] Contexts Differ
http://context-driven-testing.com/?p=38
[6] Context Driven Testing
http://context-driven-testing.com/
[7] Turning Quality on its Head
http://www.youtube.com/watch?v=cqwXUTjcabs
[8] The 10-minute Test Plan
http://googletesting.blogspot.com/2011/09/10-minute-test-plan.html
[9] RPF
http://googletesting.blogspot.com/2011/11/rpf-googles-record-playback-framework.html
[10] Career Paths using Tester Personas
http://www.stickyminds.com/s.asp?F=S16548_COL_2
