Although many training classes and conference presentations describe processes and techniques meant to help you find bugs, few explain what to do when you find a good one. How do you know what the underlying problem is? What do you do when you find a bug, and the developer wants you to provide more information? How do you reproduce those pesky, intermittent bugs that come in from customer land? In this hands-on class, Jon Bach helps you practice your investigation and analysis skills—questioning, conjecturing, branching, and backtracking. For those of you who have ever had to tell the story about the big bug that got away, Jon offers up new techniques that may trap it next time so you can earn more credibility, respect, and accolades from stakeholders. Because collaboration and participation are encouraged in this class, bring your mental tester toolkit, tester’s notebook, and an open mind.

1. TD
AM Tutorial
10/1/2013 8:30:00 AM
"The Craft of Bug Investigation"
Presented by:
Jon Bach
eBay
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Jon Bach
eBay, Inc.
With more than eighteen years of experience in software testing, Jon Bach has held technical
and managerial positions in companies including Hewlett-Packard and Microsoft. In his current
role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests
(activity flows) in eBay’s core sites to discover important bugs that threaten its core business.
He is most notable for creating, with his brother James, Session-Based Test Management, a
method to manage and report exploratory testing.

3. The Craft of (black box)
Bug Investigation
Jon Bach
QE Director, eBay
jobach@ebay.com
STAR West 2013
Preamble
These next few hours are designed with
exercises that lead you through ways of
thinking that help you narrow down a bug.
We’ll discuss techniques for bug isolation,
reproducing intermittent bugs, and skills to
help you know when you’ve gone as far as
is reasonable before handing it off.
eBay Inc. confidential
1

4. A morning of investigation
Psychic Reader
Art Show
Mysterious Spheres
Gettysbug Address
Checkout Checking Out?
Out-of-Bounds
eBay Inc. confidential
}
}
}
}
}
Ways we find bugs
CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA
Project Environment
Product Elements
Quality Criteria
Development Criteria
General Test Techniques
Customers
Information
Developer relations
Team
Equipment & tools
Schedule
Test Items
Deliverables
Structure
Function
Data
Platform
Operations
Time
Capability
Reliability
Usability
Security
Scalability
Performance
Installability
Compatibility
Supportability
Testability
Maintainability
Portability
Localizability
Function testing
Domain testing
Stress testing
Flow testing
Scenario testing
Claims testing
User testing
Risk testing
Automatic testing
eBay Inc. confidential
2

5. How *well* do we find bugs?
Some Investigation Skills and Tactics
“MR.Q COMC GOARABC R&R?”
Modeling
Chartering
Generating/Elaborating
Recording
Resourcing
Observing
Overproduction/Abandonment
Reporting
Questioning
Manipulating
Abandonment/Recovery
Collaboration
Refocusing
Alternating
Branching/Backtracking
Conjecturing
Investigation is a mindset using this skillset.
eBay Inc. confidential
How well do we investigate them?
Too Much Quality
Further time and effort
is a waste of resources.
floating line
Good enough quality bar
Further time and effort
is crucial.
Unacceptable Quality
eBay Inc. confidential
3

6. Abductive Inference
Abductive inference means finding the best
explanation for a set of data.
1.
Collect data.
2.
Find several explanations that account for the data.
3.
Find more data that is either consistent or inconsistent
with explanations.
4.
Choose the best explanation that accounts for the
important data, or keep searching.
Jump to conjectures, not conclusions
eBay Inc. confidential
Is Checkout Checking Out?
This is a dip in bids we saw in Production.
What questions do you have to troubleshoot this?
eBay Inc. confidential
8
4

7. Topics for investigation (FORCOSTTTIP)
• Feature – Do we know what happens when people bid?
• Origin -- When did this start happening?
• Recent – Did we just push code to Production?
• Consistency / Correlation -- What happened yesterday, tomorrow? Are there
other graphs and metrics?
• Overlap – Is there any other process happening?
• Security – Is it malicious? Intentional?
• Terminology -- What kinds of “bids”? US, UK, International?
• Third-Party / Silent Partner – What’s not in the room? What sort of information
have we NOT heard from?
• Truth is out there – does someone already know about this?
• Instrumentation – is the graph wrong? Can I get more data?
• Prediction – can we test a conjecture with a prediction
eBay Inc. confidential
9
Exercise: Psychic Reader
eBay Inc. confidential
5

8. Lessons
You can be psychic, too
You may be clue-less, for now
Try “defocusing”
Are you just following orders?
eBay Inc. confidential
Exercise: Mysterious Spheres
eBay Inc. confidential
6

9. Lessons
If it fails, whose fault is it?
Assumptions need nourishment
Oracles are in “control”
“Elsewhere”, is a useful thought
eBay Inc. confidential
Exercise: Out-of-Bounds
eBay Inc. confidential
7

10. Lessons
Be a “model” tester
Stagehands work behind curtains
Outside data is not inside data
Computers are our slaves
eBay Inc. confidential
Exercise: Art Show
What is my Operating Rule?
What do you notice?
What (crazy) test haven’t you tried?
eBay Inc. confidential
8

11. Lessons
“You’re so one dimensional”
Jump to *conjectures*
Yes, I *do* look FAT in this. (M vs. O)
Breaking “rules”!
eBay Inc. confidential
Exercise: Gettysbug Address
Does it work?
What is the hidden feature?
What story does the data tell?
eBay Inc. confidential
9

12. Lessons
The Rumble Strip heuristic
Missing a mission?
Be “thoughtless” (random is good)
Defocus: do *anything*, *differently*
eBay Inc. confidential
Exercise: What’s the Pattern?
Investigation of behavior is like
trying to repro a bug… what is
the behavior of this app?
eBay Inc. confidential
10

13. Lessons
Checking is different than testing
Collaboration is a kick
Not all characters are created equal
Description takes practice
eBay Inc. confidential
How well do we investigate them?
Too Much Quality
Further time and effort
is a waste of resources.
floating line
Good enough quality bar
Further time and effort
is crucial.
Unacceptable Quality
eBay Inc. confidential
11

14. How much investigation?
1) Sufficient benefits
2) No critical problems
3) The benefits outweigh the problems
4) In the present situation, and all things
considered, improvement would be more harmful
than helpful
The answer must be “Yes” to all four criteria,
but ask “for whom, what, and when?”
eBay Inc. confidential
Intermittent Bugs: Observation
Bad observation
Irrelevant observation
Bad memory
Misattribution
Misrepresentation
Unreliable oracle
Unreliable communication
Link
eBay Inc. confidential
12

15. Intermittent Bugs: System
Purposeful change, and then back to original
Accidental change
Platform change
Flaky hardware
Trespassing system
Executable corruption
Component competition
eBay Inc. confidential
Intermittent Bugs: Machine State
Frozen conditional
Improper Initialization
Resource denial
Progressive data corruption
Progressive destabilization
Overflow
Occasional functions
Different mode or option setting
eBay Inc. confidential
13

16. Intermittent Bugs: Input
Accidental
Secret boundaries or conditions
Different profile
Ghost input (alternative source)
Simultaneous action as others
Compromised input
Timing issues
Crazy Combinations
eBay Inc. confidential
Intermittent Bugs: You?
You may not be aware of...
…variables of influence
…sources of distortion in your observations
…available tools that might help
…boundaries and their characteristics
…the system's missing / extra functions
…complex / competing algorithms
eBay Inc. confidential
14

17. A few (free) investigation tools
Httpwatch
-- displays log of requests and responses
Firebug
– inspect HTML
-- JavaScript debugger
WAVE
– accessibility errors
eBay Inc. confidential
More info
• ET Dynamics:
http://www.satisfice.com/articles/et-dynamics.pdf
• Test Heuristics and Planning
http://www.satisfice.com
• Context-Driven Software Testing
http://groups.yahoo.com/group/software-testing
• Center for Software Testing Education and Research
http://www.testingeducation.org/BBST
• Books related to Exploratory Testing skills and tactics
http://www.testingreflections.com/node/view/3190
• Scenario testing examples
– http://www.testingeducation.org/a/scenario2.pdf
eBay Inc. confidential
15