Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Information overflow
1. Confidential
Information Overflow
Is too much information a bad thing?
Rev PA1 2011-12-07 1
2. Introduction
A tester’s job is to generate information and decision material for different
stakeholders
What happens if the tester generates too much information which the
stakeholder has to analyze, preventing the stakeholder from doing much
else, such as correcting defects in the case of a developer?
This is not a right/wrong situation – if the tester withholds information that is
deemed low priority or not important, that same information could be
extremely critical from a stakeholder perspective
3. Practical Example
A developer has 8 hours in a day
Each defect takes 30 minutes to analyze
Each defect takes 4 hours to fix
A tester submits 8 issues developer has time to fix 3 issues
A tester submits 24 issues developer has time to fix 1 issue
4. Triage
Of course you can have a Triage team which looks at the defects
and prioritize them for the developers, so that the most critical
defects are fixed first
The problem is that the developer still has to analyze all the defects
before the Triage team can prioritize them
5. Problem?
Information Overflow becomes a problem when the inflow of
defects is much higher than the correction rate
If the developers have to spend all their time analyzing a mountain
of defects, they will not have time to fix anything
The tester must support the developer in this task by filtering the
information they provide to their stakeholders!
6. Solution?
A tester does not provide unfiltered information to the stakeholders
either way – filtering is always performed when deciding what
defects to report, what information to include in the defects, and
what information to include in test reports and similar
When the tester filters the information to the stakeholders, the tester
must be aware of this information overflow problem, and take
actions accordingly
But how do you know which information to omit?
What can we do to solve this information overflow problem?
7. Basic Remedy
A few basic bullets that must always be taken into consideration
before any other solutions are discussed
Reports must always be accurate and understandable
Testers must understand what information their stakeholders need, and provide
that information in a usable format
Testers must understand the requirements and their priority
But even with the basics in place, how should the tester handle the
information overflow problem?
8. Information Overflow Remedy
When a defect is found, but before it is submitted to the
stakeholders, the tester should prioritize the defect using the same
model as the Triage Team would
If the defect has high priority it should always be reported
If the defect has low priority it should only be reported if the tester
believes there are available resources to fix the problem
If the tester cannot prioritize the defect, it should be submitted, and
the information gained if the defect is not fixed should be used for
future prioritizations
9. What about the low priority defects?
Information should not be forgotten – even low priority defects can
be valuable in a greater context
Even if a defect is not submitted, the information should still be
saved somewhere – low priority defects could show patterns, or
give important leads to other higher priority defects
A separate database for low priority issues that are not submitted
should be maintained – a database which most stakeholders will
never look into - but which is of great value to testers and selected
stakeholders
10. Solution Overview
Provide the right
information in the
Defect
Testers must perform these
actions
Prioritize Defect
Will be fixed Will not be
fixed
Defect Database Information Database
Testers use
Developer Analysis information for future
analysis
11. Conclusion
Testers must take the information overflow problem into account
when submitting defects and providing stakeholders with
information
A tester always filters information, but it must be done in a formal
way to combat the information overflow problem
No information should be lost, even though it is not reported to
stakeholders