1. Test Result Abstraction
Presenting Test Results with the Right Granularity

2. Introduction
• Different stakeholders have different needs when it comes to test reports
• You cannot give a developer a 1-slide power point and expect results, and
you cannot give a line manager a 40 page test report and expect that they
read it
• You also don’t know how the report will be abstracted once it leaves your
hands
• It is important to understand what information your direct stakeholders need
from you, but it is also important to understand what their stakeholders in
turn expect, so that you can try to secure that important information is easily
accessible and will not be abstracted away by mistake after the report has
left your hands
• Listen to your stakeholders, and understand what kind of information they
need, both for themselves and for their stakeholders – don’t assume that
you know what they need, and in what format they need it

3. Granularity Overview
High Management
High Management
Report
Management Report Line Management
Project Report Project Leads
Test Leads & Team Leads
Test Report A Test Report B Test Report C
Benchmarks & KPI
Engineers Found Defects Scripted Test Results
Results
Automated Regression
Exploratory Test Results Other input
Test Results

4. Hard Numbers vs. Qualitative Judgements
• To be valuable test reports need to include both quantitative and qualitative
information
• Both the actual numbers and the analysis of those numbers
• What you put front and centre depends on the stakeholders and what kind
of information they are susceptible to
• Some stakeholders will only need the qualitative judgement of the tester -
Is the product good enough?
• But others want to make that decision themselves based on abstracted
empirical information such as pass rate, number of defects, and so on
because they have other information which is relevant to the decision, such
as business or legal information
• The tester does not have to whole picture, and should not assume to know
better than their stakeholders what the stakeholders need
• It is not a testers job to decide what their stakeholders want or need, but to
provide the information that meets those wants and needs

5. A number that means one thing to an engineer, can mean
something completely different for a line manager, due to
all the external input that has been provided between the
original report and the management report
Engineers
Project Leads
Line Management
Test Leads & Team Leads
Test Report
Team Report
Project Report
Management Report
Why numbers are sometimes important
Team Information
Project Information
Business Information

6. Confidence in Results
• It is always important to provide your stakeholders with information on the
confidence of the results provided
• Zero defects found can mean that you tested nothing, or that you have a
really good product, or something in between
• Without this information, an abstraction of the results can easily be
misinterpreted
2000 TCs executed 43 Exploratory Sessions
Executed
98% Pass Rate
Good Quality Reported
Report 0 Critical Defects
0 Critical Defects
200 Man-hours 100 Man-hours
Covered all
Covered all common functionality and
use cases which features with at least
Confidence makes up for 98% of one session, focusing
regular usage of the on exceptions, error
device handling and negative
scenarios

7. Deception in Results
2000 TCs executed
98% Pass Rate
Could mean
40 Failed Test Cases
40 Failed Test Cases
28% Pass Rate for
Performance TCs 40 Critical Defects
Performance below
Several customer
customer
requirements failing
expectations
• Include as much information, with the right granularity, as necessary to
avoid this deception

8. Prioritize Information
• When writing test reports, they should be structured so that the stakeholder
can easily retrieve the most critical information they need quickly, and dig
deeper for more information if necessary
• But information that could be important to your stakeholders’ stakeholders
should also be easily accessible to secure that all relevant information
reaches its intended customers

9. Summary
• Always include both qualitative and quantitative information – when a test
report is abstracted to stakeholder higher up the corporate chain, you do not
know which information is relevant to them
• Always include enough information to avoid deception, and to give
stakeholders confidence in your test reports
• Always prioritize information in your test reports so that your stakeholders
can pick out the relevant information they need to present to their
stakeholders in turn, and to make educated decisions themselves
• Talk to your stakeholders to learn what they want and need – don’t assume
anything