How many bugs have you missed that were obvious to others? We all approach testing hampered by our own biases. Understanding our biases—preconceived notions and the ability to focus our attention—is key to effective test design, test execution, and defect detection. Gerie Owen and Peter Varhol share an understanding of how the testers’ mindsets and cognitive biases influence their testing. Using principles from the social sciences, Gerie and Peter demonstrate that you aren’t as smart as you think you are. They show how to use knowledge of biases—Inattentional Blindness, Representative Bias, the Curse of Knowledge, and others—not only to understand the impact of cognitive bias on testing but also to improve your individual and test team results. Finally, Gerie and Peter provide tips for managing your biases and focusing your attention in the right places throughout the test process so you won’t miss that obvious bug.
2. Gerie Owen
Northeast Utilities, Inc
QA Consultant Gerie Owen specializes in developing and managing offshore test
teams. Gerie has implemented the offshore model—developing, training, and
mentoring new teams from their inception. She manages large, complex projects
involving multiple applications; coordinates test teams across multiple time zones;
and delivers high quality products. Gerie’s most successful project team wrote,
automated, and executed more than 80,000 test cases for two suites of web
applications, allowing only one defect to escape into production. In her everyday life,
Gerie enjoys mentoring new test leads and brings a cohesive team approach to
testing.
3. Peter Varhol
Telerik
A testing evangelist at Telerik, Peter Varhol is a writer and speaker on software
development and quality topics. Peter has authored dozens of articles on software
tools and techniques for building applications, and has given conference presentations
and webcasts on a variety of topics—user-centered design, integrating testing into
agile development, and building the right software in an era of changing
requirements. He has held key roles on engineering teams that have produced award-
winning, quality tools such as BoundsChecker and SoftICE. Peter’s past roles include
technology journalist, software product manager, software developer, and university
professor.
5. 2/4/2014
2
Consequences of Missed Consequences of Missed
BugsBugs
• Possible Consequences of Missed Bugs:
o Negative Publicity
o Lost Sales
o Lost Customers
o Even Loss of Life
MISSED BUGS CAUSE MAYHEM
My JourneyMy Journey
The “HOW” is more important than the “WHY”The HOW is more important than the WHY
And now , I invite you to join with me
into the journey of
How Did I Miss that Bug?How Did I Miss that Bug?
6. 2/4/2014
3
How Do We Miss Bugs?How Do We Miss Bugs?
• Missed test casesMissed test cases
• Misunderstanding of requirements
• Misjudgment in risk-based testing
• Inattention
• Fatigue
• Burnout
• Multi-tasking
How Do We Test?How Do We Test?
• What is Software Testing?g
o Software testing is making judgments about
the quality of the software under test.
o Involves:
• Objective comparisons of code to
specifications,
• AND
• Subjective assessments regarding usability,
functionality etc
8. 2/4/2014
5
How do we make How do we make
judgments?judgments?
• Thinking, Fast and Slow – Daniel Kahneman
o System 1 thinking – fast, intuitive, and
sometimes wrong
o System 2 thinking – slower, more deliberate,
more accuratemore accurate
System 1 vs. System 2 System 1 vs. System 2
ThinkingThinking
• System 1 thinking keeps us functioning
o Fast decisions, usually right enough
o Gullible and biased
• System 2 makes deliberate, thoughtful
decisions
o It is in charge of doubt ando It is in charge of doubt and
unbelieving
o But is often lazy
o Difficult to engage
9. 2/4/2014
6
How Do We Apply System 1 How Do We Apply System 1
and System 2 Thinking?and System 2 Thinking?
• System 1 thinking:• System 1 thinking:
o Is applied in our initial reactions to situations.
o May employ Heuristics or rules of thumb
• System 2 thinking:
o Is applied when we analyze a problem, for example
when calculating the answer to a math problem.
S t 1 d S t 2 b i fli t• System 1 and System 2 can be in conflict:
o lead to biases in decision-making.
The Lily Pad QuestionThe Lily Pad Question
In a lake, there is a patch of lily pads. Every day, the
patch doubles in size.
If it takes 48 days for the patch to cover the entire
lake, how long would it take for the patch to cover
f fhalf of the lake?
10. 2/4/2014
7
How do Biases Impact How do Biases Impact
Testing?Testing?
• We maintain certain beliefs in testing practice• We maintain certain beliefs in testing practice
o Which may or may not be factually true
o Those biases can affect our testing results
o We may be predisposed to believe something that
affects our work and our conclusions
Ho do bias and error ork?• How do bias and error work?
o We may test the wrong things
o Not find errors, or find false errors
The Representative BiasThe Representative Bias
• Happens when we judge the likelihood of anHappens when we judge the likelihood of an
occurrence in a particular situation by how closely
the situation resembles similar situations.
• Testers may be influenced by this bias when
designing data matrices, perhaps not testing data
in all states or not testing enough types of datain all states or not testing enough types of data.
• Case Study: Completed Order bug
11. 2/4/2014
8
The Curse of KnowledgeThe Curse of Knowledge
• Happens when we are so knowledgeable aboutpp g
something, that our ability to address it from a less
informed, more neutral perspective is diminished.
• When testers develop so much domain knowledge
that they fail to test from the perspective of a new
user. Usability bugs are often missed due to thisuser. Usability bugs are often missed due to this
bias.
• Case Study: Date of Death Bug
The Congruence The Congruence BBiasias
• The tendency of experimenters to plan and executeThe tendency of experimenters to plan and execute
tests on just their own hypotheses without
considering alternative hypotheses.
• This bias is often the root cause of missed negative
test cases. Testers write test cases to validate that
the functionality works according to the
ifi ti d l t t lid t th t thspecifications and neglect to validate that the
functionality doesn’t work in ways that it should not.
• Case Study: Your negative test case or boundary
miss
12. 2/4/2014
9
The Confirmation BiasThe Confirmation Bias
• The tendency to search for and interpretThe tendency to search for and interpret
information in a way that confirms one’s initial
perceptions.
• Testers’ initial perceptions of the quality of code, the
quality of the requirements and the capabilities of
developers can impact the ways in which they testdevelopers can impact the ways in which they test.
• Case Study: Ability to print more than once bug
The Anchoring The Anchoring EEffectffect
• The tendency to become locked on and rely tooy y
heavily on one piece of information and therefore
exclude other ideas or evidence that contradicts
the initial information.
• Software testers do this often when they validate
code to specifications exclusively withoutcode to specifications exclusively without
considering ambiguities or errors in the
requirements.
13. 2/4/2014
10
Inattentional BlindnessInattentional Blindness
• Chabris and Simon conducted experiments on how
focusing on one thing makes us blind to othersfocusing on one thing makes us blind to others
o Invisible gorilla on the basketball court
o Images on a lung x-ray
Inattentional BlindnessInattentional Blindness
• a psychological lack of attention
• the tendency to miss obvious inconsistencies when
focusing specifically on a particular task.
• This happens in software testing when testers miss
the blatantly obvious bugs
14. 2/4/2014
11
WWhy Do We hy Do We Develop Develop
BiasesBiases??
• The Blind Spot Bias
o We evaluate our own decision-making process
differently than we evaluate how others make
decisions.
• West Meserve and StanovichWest, Meserve and Stanovich
Are you subject to Bias?Are you subject to Bias?
A Test Planning Exercise
This application is used to send a series of letters toThis application is used to send a series of letters to
homeowners and renters to inform them of required
inspections and get them to book appointments.
The series are different based on whether the residence is a
single or multi family and ends with a certified letter.
The bug is that if the renter to whom the initial letter was
sent moved the new renter would get the next letter in asent moved, the new renter would get the next letter in a
series, potentially a strongly-worded certified letter when
the series should start again from the first letter.
This is a medium-severity defect; but high priority as it
negatively affects customer experience.
15. 2/4/2014
12
How Does This Apply To How Does This Apply To
Testing?Testing?
W t th• We must manage the way we
think throughout the test
process.
oAs individual testersoAs individual testers
oAs test managers
oAs a professional community
How Can Testers Manage How Can Testers Manage
Their Thought Processes?Their Thought Processes?
• Use more System 1 thinking?
OR
• Use more System 2 thinking?
16. 2/4/2014
13
Test Methodology and Test Methodology and
System 2 ThinkingSystem 2 Thinking
• Test methodology is the analyticalTest methodology is the analytical
framework of testing; it invokes our system
2 thinking and places the tester under
cognitive load.
Th d t i ti f h th th t l• The determination of whether the actual
results match the expected results
becomes an objective assessment.
HHow ow DDo o WWe e FFind ind BBugs?ugs?
Focus on System 1 thinkingFocus on System 1 thinking,
intuition and emotion
17. 2/4/2014
14
Focus On System 1 Focus On System 1
ThinkingThinking
He ristics sed ith Oracles• Heuristics used with Oracles
• Recognize our emotions as indicators
of potential bugs
• Exploratory Testing
How Should We Use How Should We Use
Exploratory Testing?Exploratory Testing?
• Unstructured
o Before beginning test case execution
• Minimizes preconceived notions about
the application under test
o Oracle basedo Oracle based
• Users’ perspectives
• Data flow
18. 2/4/2014
15
How Should We Use How Should We Use
Exploratory Testing?Exploratory Testing?
• Structured
o Use to create additional test cases
• May be done earlier, possible as
modules are developed
o Session-Based
• Time-boxed charters
• Multiple testers
• Post test review session
What Can Test Managers What Can Test Managers
Do?Do?
• Foster an environment in which the testers feel
comfortable and empowered to use System 1
thinking.
o Plan for exploratory testing in the test schedule
o Encourage Testers to take risks
o Reward for Quality of bugs rather than quantity
of test cases executed
19. 2/4/2014
16
What Can the QA Profession Do?What Can the QA Profession Do?
A Paradigm Shift A Paradigm Shift
o Shift our focus from requirements coverage
based test execution to a more intuitive
approach
o Exploratory testing and business process flow
testing becomes the norm rather than the
exceptionexception
o Develop new testing frameworks where risk-
based testing is executed through targeted
exploratory testing and is balanced with scripted
testing
Question Test ResultsQuestion Test Results
• Is there any reason to suspect we are evaluating
our test results based on self-interest,
overconfidence, or attachment to past
experiences?
• Have we fallen in love with our test results?
• Were there any differences of opinion among the
team reviewing the test results?