Exploratory Testing Basics and Future

TestCon Vilnius October 19, 2017
Kari Kakkonen, Knowit Oy, FiSTB, ISTQB
@kkakkonen
Exploratory testing –
basics, experiences and future?
Knowit 1

Kari Kakkonen
ROLES
• Knowit Oy, Director/Quality and Competences, Lead Consultant, Trainer and Coach
• Treasurer of ISTQB Executive Committee
• Chairman of Finnish Software Testing Board (FiSTB)
• Auditor of Robot Framework Foundation.
ACHIEVEMENTS
• Influencing testing since 1996
• Ranked in 100 most influential IT persons in Finland (Tietoviikko magazine)
• Great number of presentations in Finnish and international conferences
• TestausOSY/FAST founding member.
• Co-author of Agile Testing Foundations book
• Regular blogger in Tivi-magazine
EDUCATION
• ISTQB Expert Level Test Management Full & Advanced Full & Agile Tester certified
• DASA DevOps Fundamentals, Scrum Master and SAFe certified
• SPICE provisionary assessor certified
• M.Sc.(Eng), Helsinki University of Technology (present Aalto University), Otaniemi, Espoo
• Marketing studies, University of Wisconsin-Madison, the USA.
BUSINESS DOMAINS
• Wide spread of business domain knowledge
• Embedded, Industry, Public,
• Training, Telecom, Commerce,
• Insurance, Banking, Pension
SERVICES
• ISTQB Advanced, Foundation and Agile Testing +
Knowit Quality Professional
• DASA DevOps Fundamentals
• Quality & Test process and organization
development, Metrics
• Agile testing, Scrum, Kanban, Lean
• Leadership
• Test automation, Mobile, Cloud, DevOps
• Quality, Cost, Benefits.
Twitter: @kkakkonen
LinkedIn: fi.linkedin.com/in/karikakkonen/
Knowit 2

Characterization of exploratory testing
• ”Planning and execution of testing is done at the same time”
(After James Bach)
• Test cases are not necessarily documented even afterwards (Cem Kaner)
• Testing is done iteratively piece by piece
• Continuous learning and interpretation of conceptions
• Utilization of knowledge gained from experience
5© Copyright Knowit Oy 2015 | Confidential

Exploratory testing - terms
• Adventure may go to sidetrack as long as you
come back to mainroad again (Kaner)
• Testing area: a bunch of functionalities
• Testing session
• Duration about ½ - 2 hours
• Time span of concentrated work is about 20
minutes
• Getting back to work takes about 20 minutes

An example: www.Topsellers.Com,
a fictional e-commerce shop
• Test case: ”Log in. Browse the content of your shopping cart. Result: Shopping cart is empty, because no
items have been picked.”
• During testing it is noticed that the shopping cart contains items! Items have been picked with the same
user account and the e-commerce shop keeps the earlier picked things in shopping cart.
• Aftertaste: ”The test should have considered this. Let’s change the test data and the test itself, and
design a new test case, which takes into account that the shopping cart can store items.”
• A familiar situation?

Learning in exploratory testing
Testing
Opinion-forming
Reporting
Designing actions
Observations
After reference: Psychology of Usability, Sinkkonen et al.

Learning from the system and customers
• What has changed or changes frequently?
• What do the customers want?
• What has been defined in an ambiguous way?
• Where do faults cluster?
• Weaknesses in the platform or programming language
• System dependencies
Reference: Lessons learned in software testing. Kaner, Bach, Pettichord

Learn from other testers, designers, from yourself
• What kind of errors do certain programmers make and how to report to and communicate with them
• What typical errors can there be in the system
• What functionalities have been built in a hurry
• What have you misunderstood and what is typically misunderstood
• How can the system be tested (especially in pair testing!)

Why exploratory testing?
Knowledge-based perspective
• Exploit the natural diversity of people in testing *)
• ”Do not plan for store”
• Systematic variation of testing
• Quick feedback to the designers intensifies learning process *)
• Spread the knowledge of a tester of a special area
*) Reference: Exploratory testing: A multiple case study. Itkonen,
Rautiainen

Testers’ approach
• Want to add more test cases and increase the coverage of the tests *)
• For defining the degree of automation of tests
• Quick overview of the quality *)
• Testing the side effects of changes – scripted testing can end up testing only the documented features *)
• The problem in regression testing is the selection of test cases, which requires user experience and
understanding of the system
• When the test documentation can not be written in advance *)
Rautiainen

Management approach
• Low management costs of test documentation
• Finding out the features of a poorly-documented component
• Optimizing the productivity of testing?
• When the available workload is limited *)
• When you want to train the customer support at the same time
Rautiainen

Where exploratory testing fits
14
Different styles of testing
Adhoc testing
Exploratory Testing
And Session-Based
Test Management
High level test cases
design and
execution
Detailed test cases
design and execution
ET + SBTM
High level TC
Detailed TC
Automation
Automated testing
and ATDD
Exploratory
Testing
ET
Systematicnessoftesting
Adhoc
ET Scripted testing

Exploratory testing as part of test strategy
15© Knowit Oy Confidental 2012
Pure (100%)
scripted testing
Scripted testing
Exploratory testing
Pure (100%)
exploratory testing
Where do you put the
focus of your own
testing?Maybe here?

Step by step
© Copyright Knowit Oy 2015 | Confidential
Plan
• Test charter
Test
session
• Notes / Log
• Bugs
Debriefing
• Dashboard

Exploratory testing in Prague – find a park
• Test charter
• ”look for green”
1819.10.2017 © Copyright Knowit Oy 2013 | Confidential | Version 1.0

• Test execution log
• Dashboard

• Defect report

Test design in a test charter
• Define the testing areas of the test object
• Divide each area to one or more test sessions
• Test charter works as a roadmap per test session
• Define test cases to be documented
• heuristic: less than 10% of all tests
• Write down test ideas and/or high level test cases

Purpose of test charter(s)
• What will be tested?
• What documents are available?
• What kind of errors are being sought?
• Tasks and what test techniques will be used?
• Targets and outputs (for example reports)
Reference: A practioner’s guide to software test design. Copeland
Ref. Exploratory testing: A multiple case study. Itkonen, Rautiainen

Charter classic example
Area Coverage
and
working
hours
Practice Documents Result possible
errors
Risks
R1. Customer’s
all selected
items are not
added to order,
Effect: 20
eur/buyer,
probability 5%
R2. Order can
not be
completed after
interruption, 5,
probability: ??
Main page 100%
Path coverage
(direct paths) and
the most common
(80% used) loops
10h
Scripting
with
Functional
Tester-tool
Main page display
description
document, navigation
map (COMING
FROM
DEVELOPMENT)
All pages and
the shopping
cart are
available
Shopping
cart
5h Shopping cart-
UC.doc (use case)
Shopping cart
can be used in
the same way
as a real
shopping cart
The same
product can
not be added
to shopping
cart several
times
Emptying the
shopping cart
causes an
execption
R1
Order ? Order-UC.doc? R2

Exploratory Testing – Test Charter another view
• Actor: intended user of the system
• Purpose: the theme of the charter including what particular objective the
actor wants to achieve, i.e., the test conditions
• Setup: what needs to be in place in order to start the test execution
• Priority: relative importance of this charter, based on the priority of the
associated user story or the risk level
• Reference: specifications (e.g., user story), risks, or other information
sources
• Data: whatever data is needed to carry out the charter
• Activities: a list of ideas of what the actor may want to do with the system
and what would be interesting to test (both positive and negative tests)
• Oracle notes: how to evaluate the product to determine correct results
• Variations: alternative actions and evaluations to complement the ideas
described under activities
Reference: ISTQB Agile Tester syllabus

Charter as an excel

Documents supporting testing
• Charter
• List of different testing strategies
• Lists of heuristics
• List of typical errors
• Kaner’s bug taxonomies*)
• Legal notices, standards, de facto-standards
• Requirements and design documentation of the system
• Self made description and models of the system behavior
• User guide *)
*) Reference: Testing Computer Software. Kaner et al.
**) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen

Kaner’s bug taxonomies:
• User interface errors
• Error handling
• Boundary-related errors
• Calculation errors
• Initial and later states
• Control Flow errors
• Errors in handling or interpreting data
• Race conditions
• Load Conditions
• Hardware
• Source and version control
• Documentation
• Testing errors
Reference: Testing computer software, p. 60 – 64

Exploratory Testing - Heuristics
• A set of heuristics can be applied when testing.
• A heuristic can guide the tester in how to perform the testing and to evaluate the results, few examples
below
Heuristics Examples
Boundaries: Approaching the Boundary e.g. almost too big, almost too
small, at the Boundary
CRUD Create, Read, Update, Delete
Configuration variations Varying the variables related to configuration e.g. Screen
Resolution; Network Speed, Latency, Signal Strength; etc.
Interruptions e.g., log off, shut down, or reboot
Source: Elisabeth Hendrickson www.testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

Performance attitude - professional working
• Keep the targets of testing in mind
• You can visit bypaths but only for a moment
• Write down observations and questions about the system
• Report in a disciplined and systematical way
• During the execution, write only the most essential test cases and in high level
• Test cases can be refined later

Tests are designed during execution
• Define a test from a question
• Design test on the basis of charter and test
ideas
• A surprising situation may indicate an error:
Utilize the surprise effect!
• Heuristics e.g.
• ”Backwards thinking”: ”This button saves
the definition text. I wonder what other ways
are there for saving the text?”

Note taking: test execution logs
• Keep a test execution log
• Keep track of the tests carried out
• Main thing is that executed tests are noted
• You may create scripted test cases of some of the tests
• Keep the most important test cases that have been executed, which show how the testing has been
done, e.g. what values have been used
• You may record your execution
• Write down also test notes for test session post-mortem and your own learning

Test charter and test log in mindmap (Xmind)

Note taking: defect logs
• In defect reporting, traceability to the requirements must be maintained so that coverage can be
evaluated
• A well written error log is the best evidence of the existence of a fault
• Report a bug clearly, so that the failure can be repeated
• You may use defect reporting systems

Testing dashboard as a test report - an example
Test area Workload Coverage Quality
level/risks
Comment
Main page !Interrupted High,
5h
Very high [all
parts + stress
tests etc..]
49: 1435, 36:
1469,
42: 1501
wait for more
pictures of
user interface
Shopping
cart
!Started
High, 2h (reserved
4h)
Low [main parts
to testing] (High)
81: 1425
[probability 9 x
effect 9; error
number. 1425]
Order !Done
6h (reserved 4h)
High [all parts]
Feedback !Not done
Low (reserved 1h)
Low [main parts
to testing]

Measuring exploratory testing
• The duration of the session
• The relative change in the number of test cases by the same tester
• Coverage of testing per session
• The number of interruptions (Suspension criteria)
• Number of rejected defects in defect database
• …
• Metrics are the eyeglasses of testing that you need in order to be fully aware of the situation and
potential problems in testing
•  It is recommended that you choose metrics that are suitable for the challenges of exploratory
testing

Some Tools
• Notetaking:
• Screenshots With Annotation
• Video with Annotation
• Intergrated bug reporting
• Test Management tool suite add-ons
• Micro Focus / HPE Sprinter
• QA Symphony qTest eXplorer
• Telerik Test Studio Explore
• qMetry Voyager
• Steps Recorder
• Recording Browser extensions
• Bug Magnet
• Notepad++ (pad tools in general)
• Xmind, Mindmup (mindmap tools in general)
• Rapid Reporter
• Test charter planning
• Charters
• Priorization
• Test Management tool suites
• Trello
• Jira Agile
• Micro Focus / HPE Agile Manager
• Pivotal Tracker
• Excel

Who to recruit? The profile of an exploratory tester
• Exploratory testing is particularly well suitable for a person, who…
• likes to take risks
• is not afraid of changes or new things
• is open-minded
• sets challenges and goals to themselves
• is smart and quick in finding test conditions
• ”Pioneer-style”
• Everyone can learn to be an exploratory tester
Reference: Choosing and Managing the Ideal Test Team. Lloyd Roden

Exploratory testing requires learning skills
• Outline the functionality of the system on paper
• Aim at understanding
• Don’t force yourself to remember facts, use documents
• Ask questions about the functionality of the system
• Recognize the items on which you need more information
Reference: Tutkiva oppiminen. Hakkarainen et al.

Improve your testing thinking skills
• Observe your own testing habits
• Recognize your own ways of thinking
• Learn from misunderstandings and mistakes
• Select the testing techniques according to the situation *)
• Improve your skills to deduce the states of the system *)
Adapted from the reference: Tutkiva oppiminen. Hakkarainen et al.,
*) Reference: Rapid Software Testing. James Bach

Exploratory future
• Exploratory testing will become the norm in agile testing
• And agile will be the norm for all development
• Testing tools suites will include exploratory testing charters and logging
• Coverage will be proven with both last quantities of well-logged exploratory testing and some formally
designed test cases
• With test automation to make sure the basics work, testers will be free to find important defects with
exploratory testing
• People will start exploring also with test automation
• Keyword driven and data-driven approaches
• AI will help with some of exploratory Testing
© Copyright Knowit Oy 2015 | Confidential

Questions?
kari.kakkonen@knowit.fi
@kkakkonen

10/19/2017 Knowit 47
Creating digital
opportunities

Exploratory Testing Basics and Future

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Exploratory Testing Basics and Future

Semelhante a Exploratory Testing Basics and Future (20)

Mais de Kari Kakkonen

Mais de Kari Kakkonen (20)

Último

Último (20)

Exploratory Testing Basics and Future