Digital QA Summit - Karen N. Johnson - Testing Explained

TESTING EXPLAINED
AN OVERVIEW OF SOFTWARE TESTING
Karen N. JohnsonDigital QA Summit

Do you find yourself needing to “check” something in production immediately after go live? Do you
need to investigate how a page or a campaign looks in multiple browsers or on a mobile device?
Without realizing it, you may be performing the role of a software tester in addition to your primary
role. What is a software tester? In this session Karen offers 1) a brief overview of the software
testing profession, 2) a summary of different types of testing, 3) a list of resources to learning more
about testing. If testing is part of what you or your team need to juggle to “make it happen” come
attend this session.

About this talk …
Testing Explained © Karen N. Johnson 2014

•  Software Test Consultant
•  Published Author (Beautiful Testing)
•  Co-founder of WREST: Workshop on Regulated Software Testing
•  Website:
www.karennicolejohnson.com or www.karennjohnson.com
•  Twitter: @karennjohnson
Karen N. Johnson

Hey, can you “just” check this out?

Can you make sure all the parts to the campaign “look good?”


But do you feel unsure how to test?


to software testing

Testing is more than checking things out.

Software testing is a profession with it’s own “world.”

When you already have a fulltime job and a profession?
But how do you immerse yourself in another world?

Here’s look at
some of the essential topics
in the software testing community.

Waterfall & Agile Development
¨  There are many different ways to develop and test
software.
¨  In past years, the Waterfall method was the most
common software development methodology.
¨  In current times, an Agile development process is
now more common.

Waterfall Development

Agile Development
The Agile Manifesto
¨  We are uncovering better ways of developing software by
doing it and helping others do it. Through this work we have
come to value:
¤  Individuals and interactions over Processes and tools
¤  Working software over Comprehensive documentation
¤  Customer collaboration over Contract negotiation
¤  Responding to change over Following a plan

Acronyms: we love them!
¨  ATDD: Acceptance Test-Driven Development
¨  BDD: Behavior-driven development
¨  TDD: Test-driven development

Checking vs. Testing
¨  Within the profession of testing, there is a long-time discussion known as “checking
vs. testing.”
¨  Often less experienced people in testing and/or people who are not professional
testers “test” in such a manner that they are “checking” software works.
¨  In these situations, people are not exploring, they are not looking for issues, they are
checking to confirm software works versus looking to expose or discover issues.

¨  If you never read an article on testing, if you never read a blog post on
checking vs. testing, learn this one concept:
Know when you are checking to confirm software is working vs. testing with the
intention of discovering possible issues.
¨  Do not confuse the two approaches (checking vs. testing) as meaning the same
thing.
¨  There is nothing wrong with “checking” software or “confirming” that software
works but this is not the same as intentionally looking to discover defects.

Tested = Checked + Explored
In Elisabeth Hendrickson’s book, Explore It!, she states that neither checking nor
exploring is enough.
“You’re not done testing until you’ve checked that the software meets expectations
and you’ve explored whether there are additional risks. A comprehensive test
strategy incorporates both approaches.”

Exploratory vs. Scripted Testing
¨  Within the profession of testing, there is a long-time
discussion on the use of scripted vs. exploratory
testing.
¨  Let’s look at each format.

Scripted testing: a definition
Scripted testing follows a path that is written by the tester
themselves or someone else. The script includes test steps that are
documented. It is expected that the tester will follow the steps as
detailed. The tester’s job in a scripted environment is to follow
each step and report findings (if any).

Exploratory testing: a definition
James Bach’s definition:
“Exploratory testing is simultaneous learning, test design, and test execution.”
This style of testing requires that your brain be fully engaged at all times.
Cem Kaner’s definition:
“Exploratory software testing is a style of software testing that emphasizes the
personal freedom and responsibility of the individual tester to continually optimize
the value of her work by treating test-related learning, test design, test execution,
and test result interpretation as mutually supportive activities that run in parallel
throughout the project.”

Exploratory & Scripted Testing
¨  As it turns out, you don’t have to choose one form
over another. You could use a blended approach.

Schools of Software Testing
¨  Around 2003, several people in the testing field constructed a
way to explain different core approaches to software testing. The
construct was referred to as the schools of testing.
¨  The schools of testing have been greatly debated over the years.
Some people have been so opposed to the construct of the
schools concept, they have refused to even discuss the construct or
acknowledge the schools as a way of discussing differing views.

¨  In 2003, the schools were defined by Bret Pettichord as:
¤  Analytical School
¤  Factory School
¤  Quality Assurance
¤  Context-Driven School

¨  You don’t have to be involved in the debate.
¨  You don’t have to choose a school.
¨  But it can be helpful to know this “argument” exists in the testing profession.
¨  It can be helpful to know what school a tester identifies with.

The Seven Basic Principles of the Context-Driven School
1.  The value of any practice depends on its context.
2.  There are good practices in context, but there are no best practices.
3.  People, working together, are the most important part of any project’s context.
4.  Projects unfold over time in ways that are often not predictable.
5.  The product is a solution. If the problem isn’t solved, the product doesn’t work.
6.  Good software testing is a challenging intellectual process.
7.  Only through judgment and skill, exercised cooperatively throughout the entire
project, are we able to do the right things at the right times to effectively test
our products.

The Original Quadrants of Testing
From Brian
Marick’s blog

The Original Quadrants of Testing
Support programming vs. critique product
¤  “When it comes to supporting programming, tests are
mainly about preparing and reassuring.”
¤  “On the other side of the division, tests are about
uncovering prior mistakes and omissions.”

Agile Testing Quadrants
From the
book, Agile
Testing by Lisa
Crispin and
Janet
Gregory

The Quadrants are being reconsidered
¨  Gojko Adzic
http://gojko.net/2013/10/21/lets-break-the-agile-testing-quadrants/
¨  Markus Gärtner
http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/
¨  Elisabeth Hendrickson
http://www.slideshare.net/ehendrickson/the-thinking-tester-evolved?ref=http://
www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/

Certification & Standards
¨  There are several certification programs available.
¨  There are several boards such as IEEE and ISO, that
devise standards around testing.
¨  And there are testers who are in agreement with
certifications and/or standards.
¨  And there are testers who are strongly opposed.

Types of Testing
¨  There are numerous types of testing. Each type has a focal point and serves
a different purpose. For example, user acceptance testing vs. performance
testing.
¨  In some cases, a type of testing will be more likely to occur at different
times during the development process. For example, unit testing vs. system
testing.

Black Box Testing
¨  Black box testing is one of the most common terms used in testing.
¨  The basic premise is the person testing does not know how the code
was written and does not have insight inside the “box” or the system.
¨  Black box testing is testing without a technical understanding of how
a program was written. This is why the “box” (or the item being
tested) is “black” (because we don’t know what is inside.)

White Box or Glass Box Testing
¨  The opposite of black box testing. White box testing means the tester has
an inner understanding of how the program was written, this form of testing
allows for more targeted technical testing.
¨  Much of the time, black box testing is the type of testing being used.

An example …
Suppose I am testing the credit card entry part of an ecommerce website.
Black box testing:
•  Try using an invalid credit card
White box testing:
•  Review the code
•  Learn after three invalid credit card entries, the account is locked
•  Enter three invalid attempts check error handling and messaging
With black box testing, I might try an invalid credit card once, see an error message
and not continue to test because I don’t know about the three attempt condition.

Unit Testing
Martin Fowler’s description:
“The common properties of unit tests — small scope, done by the programmer
herself, and fast — mean that they can be run very frequently when
programming.”

Integration Testing
Paul Jorgensen’s definition:
Integration is the phase in software testing in which individual software
modules are combined and tested as a group. It occurs after unit testing.
Lee Copeland’s definition:
In integration we assemble units together into subsystems and finally into
systems. It is possible for units to function perfectly in isolation but to fail when
integrated.

System Testing
A system consists of all of the software (and possibly hardware, user manuals,
training materials, etc.) that make up the product delivered to the customer.
System testing focuses on defects that arise at this highest level of integration.
Typically system testing includes many types of testing: functionality, usability,
security, internationalization and localization, reliability and availability,
capacity, performance, backup and recovery, portability, and many more.

UAT or User Acceptance Testing
Acceptance testing is defined as that testing, which when completed
successfully, will result in the customer accepting the software and giving us
their money.

An example …
Suppose I am testing the purchasing process of an ecommerce website.
Unit testing:
Build a unit test for credit card expiration date.
Integration testing:
When the code for credit card processing is integrated with the code that handlings
shipping information, I can build an integration test.
At the system level, I might build testing from the start of product selection through
the entire place an order, credit card and shipping process; a test that brings all the
pieces together.

Regression Testing
¨  A common form of testing; used to retest to ensure
changes in one part of a program have not
introduced issues in another part of a program.

The “ilities” of testing
¨  Software testers commonly refer to a series of terms as the
“ilities” of testing.
¨  The “ilities” are often a challenge to test but are often the
same items that customers are often most concerned about.

The “ilities” of testing
¨  Capability
¨  Reliability
¨  Usability
¨  Security
¨  Scalability
¨  Performance
¨  Installability
¨  Compatibility
¨  Supportability
¨  Testability
¨  Maintainability
¨  Portability
¨  Localizability
Testing Explained © Karen N. Johnson 2014 Source: James Bach, Rapid Software Testing

More types of testing
¨  Stress
¨  Multi-user
¨  Cookies
¨  Compatibility
¨  Developer
¨  Unit
¨  Integration
¨  System
¨  User Acceptance
¨  Installation
¨  API/web services
¨  Black box
¨  White or glass box
¨  Gray box
¨  Functional
¨  Automated
¨  Regression
¨  Security
¨  Data
¨  Exploratory
¨  Scripted
¨  Performance

Some “words” around testing
¨  Risk
¨  Heuristics
¨  Oracles

But if you only learn one question to
ask, let the question be …
What if?

•  What if? Is a challenge question.

•  What if? Is the type of question that begs for investigation and
discovery.

•  What if? Does not require technical training or knowledge.


During storyboarding and
design is a great time to
use: What if?


Testers focus on where

and how a story

can breakdown.


“State transitions provide a

Bermuda-Triangle-like moment in time
where “things” can go wrong for a user
or system. And once a user or system
is in a bad state, they can be awkwardly
trapped or lost.”


State transition
“When a system must remember something about what has happened
before or when valid and invalid orders of operations exist, state-
transition diagrams are excellent tools to record this information.”
Source: Lee Copeland. A Practitioner's Guide to Software Test Design

1. You will never find everything.
The number of combinations and possibilities is greater than what you can test.

There are more ways to test than there are hours in the day.
2. You will never have enough time.

If you have more ideas for testing than time, choose what you will test wisely and get
help from your team.
3. You cannot do it all alone.

Build testing into the job regardless of who in your agency gets it done. Do not allow
testing to be an optional activity.
4. Don’t allow testing to be an extra bill to the client.

Note-taking, mind mapping, voice recording: whatever it takes, don’t lose a great idea.
5. Capture your test ideas.

6. To find issues, you have to test beyond the simple path.

7. Ask questions. Especially ask: “what if?”

Email: karen@karennjohnson.com Twitter: @karennjohnson
Karen N. Johnson

¨  James Bach, Blog: Testing and Checking Refined,
http://www.satisfice.com/blog/archives/856
¨  Michael Bolton, Blog: Testing vs. Checking
¨  http://www.developsense.com/blog/2009/08/testing-vs-checking/
¨  Elisabeth Hendrickson, Explore It!
¨  Ministry of Testing,
http://www.ministryoftesting.com/2012/07/mindmaptesting-and-checking/

Exploratory vs. Scripted Testing
¨  James Bach, paper: Exploratory Testing Explained (10pp),
http://www.satisfice.com/articles/et-article.pdf#page=1&zoom=auto,-55,792
¨  James Bach, video, Understanding exploratory testing (6 minute video),
https://www.youtube.com/watch?v=I-ItEKt_N_s
¨  James Bach, Session-Based Test Management (1 page), http://www.satisfice.com/sbtm/
¨  James Bach, What is Exploratory Testing? And How it Differs from Scripted Testing
¨  http://www.satisfice.com/articles/what_is_et.shtml
¨  Cem Kaner, slide set (180pp) A Tutorial in Exploratory Testing,
http://www.kaner.com/pdfs/QAIExploring.pdf
¨  Michael Bolton, Resources on Exploratory Testing, Metrics, and Other Stuff,
http://www.developsense.com/resources.html
¨  Gregory Mooney, Exploratory vs. Scripted Testing: One or the Other, or Both?,
http://blog.smartbear.com/testing/exploratory-vs-scripted-testing-one-or-the-other-or-both/

Schools of Testing
¨  Bret Pettichord, Schools of Software Testing
https://www.prismnet.com/~wazmo/papers/four_schools.pdf
¨  Bret Pettichord,
http://www.testingeducation.org/conference/wtst_pettichord_FSofST2.pdf
¨  Cem Kaner, Blog: Schools of software testing, http://kaner.com/?p=15
¨  Context-driven testing, http://context-driven-testing.com/
¨  James Bach, http://www.satisfice.com/blog/archives/134

Quadrants of Testing
¨  Brian Marick blog post:
http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2
¨  Lisa Crispin, Janet Gregory, Agile Testing (book)
¨  Gojko Adzic
http://gojko.net/2013/10/21/lets-break-the-agile-testing-quadrants/
¨  Markus Gärtner
http://www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/
¨  Elisabeth Hendrickson
http://www.slideshare.net/ehendrickson/the-thinking-tester-evolved?ref=http://
www.shino.de/2012/07/30/the-testing-quadrants-we-got-it-wrong/

Rapid Testing
¨  Rapid testing is a method, a series of techniques and a class,
all in one taught by James Bach.
¨  James is arguably the most known tester across the globe.
¨  James posts the slides to his Rapid Software Testing class on his
website. www.satisfice.com

BBST: the Black Box Software Testing
¨  Cem Kaner offers a series of classes taught through slides and
videos.
¨  Cem posts the materials to his classes:
http://www.testingeducation.org/BBST/

Digital QA Summit - Karen N. Johnson - Testing Explained

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Digital QA Summit - Karen N. Johnson - Testing Explained

Semelhante a Digital QA Summit - Karen N. Johnson - Testing Explained (20)

Mais de Karen N. Johnson

Mais de Karen N. Johnson (10)

Último

Último (20)

Digital QA Summit - Karen N. Johnson - Testing Explained