Exploratory Testing Is Now in Session

MM
PM Tutorial
9/30/2013 1:00:00 PM

"Exploratory Testing Is Now in
Session"
Presented by:
Jon Bach
eBay

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Jon Bach
eBay, Inc.
With more than eighteen years of experience in software testing, Jon Bach has held technical
and managerial positions in companies including Hewlett-Packard and Microsoft. In his current
role as director of Live Site Quality for eBay, Jon is dedicated to building “end-to-end” tests
(activity flows) in eBay’s core sites to discover important bugs that threaten its core business.
He is most notable for creating, with his brother James, Session-Based Test Management, a
method to manage and report exploratory testing.

Exploratory Testing:
Now in Session

Jon Bach
QE Director, eBay
jobach@ebay.com
eBay Inc. confidential

STAR West 2013

Testing is about DISCOVERY…


1

…finding emerging context


enhanced by conference…


2

...affected by inference


…framed by reference


3

Key Idea

Exploratory testing
is a combination of

Imagination
and

Discovery

}

}

}

}

}

How to Discover with Your Imagination

CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA
Project Environment

Product Elements

Quality Criteria

Development Criteria

General Test Techniques

Customers
Information
Developer relations
Team
Equipment & tools
Schedule
Test Items
Deliverables

Structure
Function
Data
Platform
Operations
Time

Capability
Reliability
Usability
Security
Scalability
Performance
Installability
Compatibility

Supportability
Testability
Maintainability
Portability
Localizability

Function testing
Domain testing
Stress testing
Flow testing
Scenario testing
Claims testing
User testing
Risk testing
Automatic testing


4

“The” ET Definition

A style of software testing…
that emphasizes the personal freedom…
and responsibility of the individual tester…
to continually optimize the quality of his/her work…
by treating test-related learning…
test design…
test execution…
and test result interpretation…
as mutually supportive activities…
that run in parallel…
throughout the project.
-- Cem Kaner, 2006


The agile nature of exploration and the ability
of testers to rapidly apply their skills and
experience make exploratory testing a widely
used test approach—especially when time is
short.
But exploratory testing is often dismissed by
project managers who assume that exploratory
testing is not reproducible, measurable, or
accountable.


5

Why this tutorial?

1) Exploratory testers want respect: When testers explore
during testing, they find great bugs. However, since they
often don’t know how to describe their thinking, it’s
considered to be dismissed as “playing around”.
2) The documentation dilemma: Project managers may insist
that all testing be documented, but Agile does not emphasize
that. So how to balance time spent documenting with time
spent testing?
3) Your work might be scrutinized: You may have to give a
report someday about something you did that was
exploratory – like attending this conference.

My missions for you

1) Discuss — and practice — a way to manage
and measure exploratory testing.
2) See some free tools that support sessionbased exploration.
3) Learn why exploratory testing can be
thoughtful, structured, and measurable.
4) Have some fun.

6

A structure for exploration

Mission
Tactics
Story


Exploratory exercise

Test this…


7

Scrutiny
What’s being asked

What they may be thinking

What was your mission?

Remind me what I told you to do.
Why did you do that?

How did it go?

Were you careful or reckless?
What should I be worried about?

How far did you get?

Are we closer to shipping?
Can you help me know our status?

Need anything?

Can I speed this along?
Do you need more of my time?

When will you be done?

Will I get my bonus?
I have a new task for you…


How could you accurately report your testing?


8

Session-Based Exploration

The “Session”
1)
2)
3)

Time Box
Reviewable Result
Debriefing


Time Box: focused effort, fixed duration

Short: 60 minutes (+-15)
Normal: 90 minutes (+-15)
Long: 120 minutes (+-15)
– Brief enough:
• for accurate reporting
• to allow flexible scheduling
• to allow course correction

– Long enough:
• to get solid testing done
• for efficient debriefings


9

The deliverable: a session report

• Charter
– #AREAS

CHARTER
----------------------------------------------Analyze MapMaker’s View menu functionality and
report on areas of potential risk.

• Notes

#AREAS
OS | Windows 2000
Menu | et-jsb-010417-c.sesView
Strategy | Function Testing
Strategy | Functional Analysis
Sample
START
----------------------------------------------5/30/00 03:20 pm
TESTER
----------------------------------------------Jonathan Bach
TASK BREAKDOWN
----------------------------------------------#DURATION
short





Metrics
–
–
–
–
–

Bugs
–

#TEST DESIGN AND EXECUTION
65
#BUG INVESTIGATION AND REPORTING
25

#BUG

#SESSION SETUP
20

#DURATION
#SESSION SETUP
#CHARTER / OPPORTUNITY



Issues
–

#ISSUE


Anticipating Scrutiny
Agenda: “PROOF”
Past
Results
Obstacles
Outlook
Feelings

Past: What kinds of things did you do?
Results: What were your findings?
Obstacles: What slowed your mission?
Outlook: Where do we go from here?
Feelings: How are you feeling about what happened?

10

Session report

My session write-up from exploratory testing for
Agile2010…
CHARTER
#AREAS
.LINK
OS | Windows 2000
Menu |
START
----------------------------------------------5/30/00 03:20 pm
TESTER
----------------------------------------------Jonathan Bach
TASK BREAKDOWN
----------------------------------------------#DURATION
short
65
25
#SESSION SETUP
20


eBay-related sample

CHARTER
----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of
potential risk.
#AREAS
LINK
START
----------------------------------------------5/30/00 03:20 pm
TESTER
----------------------------------------------Jonathan Bach
TASK BREAKDOWN
----------------------------------------------#DURATION
short
65
25
#SESSION SETUP
20"C:Documents and SettingsjobachDesktopLive
Siteisrat_sessionsSession_Listing_Promo.txt"


11

Exercise:

An “exploration and analysis” session


When to use SBTM
Getting support: “We have thousands of test cases and a
huge regression suite, but some of our best bugs come from ad
hoc testing. Why is that?”
Improving skill: “How she finds those great bugs without test
cases, I’ll never know. I guess some people are just natural
explorers -- you either have it or you don’t -- and I just don’t
have her knack for it.”
Earning trust and autonomy: “The triage team needs more
information about these 3 bugs, can you give me that by the
end of the day?”


12

First: what’s there to test?


Second: ideas for charters
• What item for sale is nearest to you location?
Submitted the most and sold the least?
What item appears most with no bids?
What are the most recent items posted for any given category?
Which category has the most items?
Which search strings create null queries?
What are 3 examples of broad queries vs 3 examples of ambiguous queries?
What queries create invalid results?
How do you find "sold" items?
What is the most common item?
What is the most bizarre item?
What is the most expensive item?
Is it possible to restrict results to one category without the categories field?
Which seller has listed the most items which have not sold?
What search string takes the longest to return results?
Is there a search that works differently on different browsers?
What was the highest bid for an item in support of any charity?
Find an item that the seller has no business selling!
Find an item you find only on eBay, not Amazon.
What item could you find that had the most bids?
How many eBay stores are there?
What is the oldest object (not item) for sale on ebay?
Are results different if you are logged in?
Try a search on ebay.com and then try the same on on your mobile. What are the 3 major
differences?
What item has the greatest difference between the highest and second highest bids?
What user has bid the most but won the least?
What item has been on eBay the longest? Is there a way to tell the age of the item ID?
Who has sold the most items?
Is there a way to find the most commonly misspelled item?
What item can be found in the most categories?

13

Sources of charters
• Documentation / Specifications
• Web forums
• Previous products

• Team members
• Competing products
• Your expertise
• Help files
• Manuals

High-level sources

Test Plan Evaluation Model
http://www.satisfice.com/tools/tpe-model.pdf

Test Planning Checklist
http://www.satisfice.com/tools/build-the-plan.pdf

Heuristic Test Strategy Model
http://www.satisfice.com/tools/satisfice-tsm-4p.pdf


14

Charters
• General charters may be necessary at first:
• “Analyze the Insert Picture function”
• Specific charters provide better focus, but take more
effort to design:
• “Test clip art insertion. Focus on stress and
flow techniques, and make sure to insert into a
variety of documents. We’re concerned about
resource leaks or anything else that might
degrade performance over time.”

The power of collaboration

As a London Underground rider,
I want to enter my ticket into the kiosk,
Because I want to go to Heathrow.

What are the possible outcomes?

http://www.faqs.org/faqs/uk/transport-london/section-6.html


15

The session report

• Charter
– #AREAS

CHARTER

• Notes

#AREAS
OS | Windows 2000
Sample
START
----------------------------------------------5/30/00 03:20 pm
TESTER
----------------------------------------------Jonathan Bach
TASK BREAKDOWN
----------------------------------------------#DURATION
short



65



Metrics
–
–
–
–
–

25

Bugs
–

#BUG

#SESSION SETUP
20

#DURATION

#SESSION SETUP

Issues
–

#ISSUE


A boring (Boeing) story


16

On October 30, 2006, Alaska Airlines flight 61, a Boeing
737-200, N740AS, took off from runway 34 right at
Seattle-Tacoma International Airport, Seattle,
Washington.
There were no injuries to the 71 passengers or 5
crewmembers and there was no damage to the airplane.
The airplane was operating under the provisions of 14
CFR Part 121 and continued uneventfully to its
destination of Juneau International Airport, Juneau,
Alaska.

The Report


17

Why the report?
1. The government is collecting statistics on runway
incursions, which are potentially deadly.
2. Since the pilots or ground controllers are subject to
reprimand or dismissal for this sort of thing, a legal record that
it happened is necessary.
3. Ground controllers or signage may have caused the
problem. You need a record of the event, and
investigation, to discover if there is another accident
waiting to happen.


Styles I’ve seen

1) Novel Narrative

2) Flight Data Recorder

3) ESPN Highlight


18

Clark’s notes…
Captain Clark, River Dubois opposite the mouth of the
Missouri River, 13 May 1804
“I dispatched an express this morning to Captain Lewis
at St. Louis. All our provisions, goods, and equipage on
board of a boat of 22 oars [party], a large pirogue of 71
oars [in which 8 French], a second pirogue of 6 oars
[soldiers], complete with sails, &c. Men completed with
powder cartridges and 100 balls each, all in health and
readiness to set out. Boats and everything complete, with
the necessary stores of provisions and such articles of
merchandise as we thought ourselves authorized to
procure-though not as much as I think necessary for the
multitude of Indians through which we must pass on our
road across the continent.”

What to write while exploring
Historical Explorer

Tester

Observations

•

drawings of flora / fauna
descriptions of indigenous people
• landmarks

•

(To the degree you think they are
relevant to stakeholders)

•

•

feature model
text from log files
• text from dialogs

Conjectures

•

what is this thing?
where should we go today?
• how do we get there?
• new orders from HQ?
• are those people hostile?

•

(Inferences based on experiences.
After I test, I think I know something)

•

•

Project information

•

mission
supplies and staff
• latitude / longitude
• death and disease
• supply status

•

(Independent of observer)

•

•

test ideas
questions
• product and project
issues
• concerns
• risks
charter
test actions
• config info
• build details
• tools used


19

Scrutiny: what happened here?
“(Orderly Book) Camp Mouth of the Kansies June 29, 1804
Ordered --- A Court Martiall will Set this day at 11 oClock... for the
trial of John Collins and Hugh Hall...
John Collins Charged "with getting drunk on his post this Morning
out of whiskey put under his charge as a Sentinal, and for suffering
Hugh Hall to draw whiskey out of the Said Barrel intended for the
party."
...The commanding Officers approve of the Sentence of the Court
and orders that Punishment take place at half past three this
evening, at which time the party will Parrade for inspection.”
June 29, 1804
William Clark


Jon Bach, 2001

Sample test notes from actual exploratory testing :
et-jsb-010416-a.ses
et-jsb-010416-b.ses (for Issues)
et-jsb-010416-c.ses (a map for more ideas)
et-jsb-010417-a.ses (Claims Testing)
et-jsb-010417-b.ses (for Bugs)
et-jsb-010417-c.ses (for referenced notes)
et-jsb-010418-c.ses (more Claims)


20

How much detail is enough?

1) Sufficient benefits
2) No critical problems
3) Benefits outweigh problems
4) All things being equal, further documentation (or time,
effort, testing, editing, etc.) is more harmful than helpful

All 4 conditions must be met to know when you’re done,
(From brother James’ Rapid Software Testing course)

But ask: “good enough for who, what and when?”

Metrics and Mechanics

Gathering empirical data about the object of your study;
collecting different kinds of data, or data about different
aspects of the object. Designing experiments and establishing
lab procedures.


21

The session report
• Charter
– #AREAS

CHARTER

• Notes

#AREAS
OS | Windows 2000
Sample
START
----------------------------------------------5/30/00 03:20 pm
TESTER
----------------------------------------------Jonathan Bach
TASK BREAKDOWN
----------------------------------------------#DURATION
short



65



Metrics
–
–
–
–
–

25

Bugs
–

#BUG

#SESSION SETUP
20

#DURATION

#SESSION SETUP

Issues
–

#ISSUE


What to measure

Bug Investigation
Test Design and Execution

(and Reporting)

Session Setup


22

Reporting the metrics

• Test, Bug, and Setup are orthogonal categories, but all we’re only tracking
interruptions to testing

• Nearest 5% or 10% is good enough

• If activities are done simultaneously, report the highest precedence activity:
T, B, then S


TBS Report
Test Design & Execution
Bug Investigation
Environment Config
Automation (BLOCKERS)
Learning-Gathering Requirements

15%
35%
15%

10%


25%

23

Sample project

HTML status report


Tools

1)

Scan Tool

2)

Session Tester (Jonathan Kohl)

3)

Rapid Reporter (Shmuel Gershon)

(James & Jon Bach)


24

Diagnosing productivity

Test
28%

• Do these proportions make
sense?

Bug
4%

• How do they change over time?
• Is the reporting protocol being
followed?

Setup
6%

Non-Session
61%

Opportunity
1%

300.0
250.0
200.0
150.0
100.0
50.0
0.0 Inc. confidential
eBay
5/26
6/9

6/23

7/7

7/21

8/4

8/18

Estimating a Test Cycle

1. How many perfect sessions (100% on-charter testing)
does it take to do a cycle? (let’s say 40)
2. How many sessions can the team (of 4 testers) do
per day? (let’s say 3 per day, per tester = 12)
3. How productive are the sessions? (let’s say 66% is

on-charter test design and execution)

4. Estimate: 40 / (12 * .66) = 5 days
5. We base the estimate on the data we’ve collected.
When any conditions or assumptions behind this
estimate change, we will update the estimate.

25

Testing is like looking for words…
???

(10 minutes)


Debriefing
Mission A: What happened during the session?
Mission B: How many words did you find?
Mission C: What obstacles did you face?
Mission D: Do you think you found all the words?
All groups: How did you feel about your mission?
Agenda: “PROOF”
Past
Results
Obstacles
Outlook
Feelings


26

Next steps?

1) Tell the “tactics” story: Practice using terminology for the
skills used during exploration. It has helped managers
understand the sapience and discipline of exploration.
2) Try a three-week SBTM pilot: Frame your explorations in
sessions (time-boxed effort with a report) and see how your
stakeholders react.
3) Use PROOF to anticipate scrutiny: Testing ourselves is just
as important as testing software. It has won testers more
credibility, autonomy, and respect.


More info
• ET Dynamics:
http://www.satisfice.com/articles/et-dynamics.pdf
• Test Heuristics and Planning
http://www.satisfice.com
• Context-Driven Software Testing
http://groups.yahoo.com/group/software-testing
• Center for Software Testing Education and Research
http://www.testingeducation.org/BBST
• Books related to Exploratory Testing skills and tactics
http://www.testingreflections.com/node/view/3190
• Scenario testing examples
http://www.testingeducation.org/a/scenario2.pdf


27

Exploratory Testing Is Now in Session

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (8)

Mais de TechWell

Mais de TechWell (20)

Último

Último (20)

Exploratory Testing Is Now in Session