Elisabeth Hendrickson’s book, Explore It!, contains this definition: “Tested = Checked + Explored”. When I read it, I was fascinated. “What does that mean?”, I asked myself, “what does it /really/ mean?” This talk described the journey I undertook to understand it, and other definitions of testing that I found along the way, and then to come up with a new definition that filled the gaps I saw in the others, without losing the aspects of them that I felt were valid and useful. Essentially, I formalised what testing is for me. And, now that I have my definition, I can ask myself in any given situation whether my actions are consistent with the way I believe I want to behave.

1. Linguamatics, UK
James Thomas
The Anatomy of a Definition of Testing
@qahiccupps

2. Testing
is the pursuit of relevant incongruity
Testing (for me)
is the pursuit of relevant incongruity
@qahiccupps

3. Tested = Checked + Explored
Neither checking nor exploring is sufficient on its own
… interact with the software or system, observe its actual
behavior, and compare that to your expectations
@qahiccupps

4. Does testing have to be on the software?
“Tested = Checked + Explored” …
… but for what values, for what units, and what does “+” mean?
Past tense suggests testing and finishing testing.
Do we always need both checking and exploring?
How does this relate to the testing/checking distinction?
@qahiccupps

5. Was I testing?
reading
cross-referencingnote-taking
cataloguing
correlating
analysing
sense-making
skim-reading
reasoning
searching
re-reading
reflecting
researching
factoring
filtering
comparing
model-building
consistency-seeking
thinking laterally
thinking critically
being sceptical
evaluating
I criticised the definition
I made and challenged models
I analysed Elisabeth’s answers
I reflected on the way I asked questions
I wondered whether any of this was relevant
@qahiccupps

6. But was I testing?
It felt like testing!
@qahiccupps

7. Some definitions of testing
“Testing is the process of executing a program with the
intent of finding errors.”
Glenford J. Meyers; The art of software testing [1979]
“The penultimate objective of testing is to gather
management information.”
Boris Beizer, Black Box Software Testing: Techniques for
Functional Testing of Software and Systems [1995]
“Software testing is the dynamic execution of software
and the comparison of the results of that execution
against a set of pre-determined criteria.”
NIST The Economic Impacts of Inadequate Infrastructure
for Software Testing [2002]
“Testing is… an infinite process of comparing the invisible
to the ambiguous in order to avoid the unthinkable
happening to the anonymous”
Jon Bach,Telling Your Testing Story, SEASPIN [2008]
“Testing is the process of evaluating a product by learning
about it through exploration and experimentation, which
includes to some degree: questioning, study, modeling,
observation, inference, etc.”
James Bach, Michael Bolton [2013]
Arborosa,
What is software testing? @qahiccupps

8. Testing is simple: you understand what is
important and then you test it
Adam Knight, Fractal Exploratory Testing
Rikard Edgren, EuroSTAR 2015
as each flaw ... is discovered ... [a] mini
exploration will result in a more targeted
testing exploration around [the] feature area
@qahiccupps

9. Testing a sub-sub-feature
Testing a sub-feature
Testing a feature
Testing the model
Testing the PO’s view
Testing the links between features
Testing another feature
Testing PO’s expression of their view
Testing the way I’m talking to the PO
Testing whether the PO is the best
person to talk to
Testing the end user need
Testing the feature testing
Testing the reason for testing
@qahiccupps

10. @qahiccupps

11. John Bach,
Telling Your Testing Story @qahiccupps

12. So what is testing about (for me)?
Often non-linear
About more than software
About potential issues
Looking and maybe finding
Takes place in a context, at a time, for a reason
Takes place to inform a decision
@qahiccupps

13. Testing
is the pursuit of relevant incongruity
Testing
is the pursuit of relevant incongruity
@qahiccupps

14. Testing is the pursuit of relevant incongruity
not in harmony or keeping
@qahiccupps

15. Testing is the pursuit of relevant incongruity
finding
exploring
investigating
@qahiccupps

16. Testing is the pursuit of relevant incongruity
to someone who matters
@qahiccupps

17. Testing
is the pursuit of relevant incongruity
Testing
is the pursuit of relevant incongruity
@qahiccupps

18. The goal of testing is identifying problems that matter
Michael Bolton, Quality Remarks podcast #10
@qahiccupps

19. Testing
is the pursuit of relevant incongruity
Testing (for me)
is the pursuit of relevant incongruity
@qahiccupps

20. Linguamatics, UK
James Thomas
The Anatomy of a Definition of Testing
@qahiccupps

21. Selected References
Jon Bach: https://www.seaspin.org/s/telling_your_testing_story.pdf#page=11
Rikard Edgren:http://qahiccupps.blogspot.co.uk/2015/11/testing-is-simple-and-complicated.html
Adam Knight: http://www.a-sisyphean-task.com/2013/01/fractal-exploratory-testing.html
Arborosa: https://arborosa.org/2015/06/14/what-is-testing/
Quality Remarks: http://qualityremarks.com/qr-podcast/
Explore It!: https://pragprog.com/book/ehxta/explore-it

22. 22
Some Comments
Helicopter view
• it is sufficiently open that people could buy into it, and read into it, particularly non-testers.
• it's accurate and to the point.
• it has the feel of Weinberg's definition of a problem.
• it sounds profound but I'm not sure whether there is any depth.
• it seems very close to the regular notion of targeting information/unknowns.
Coverage
• can not testing be part of this idea of testing?
• Cem Kaner talks about balancing freedom and responsibility in testing. Is that covered here?
• the definition doesn't talk about risk.
Practical utility
• it couldn't be used to help someone new to testing decide what to do when testing.
• I could imagine putting this onto a sticky and trying to align my actions with it.
Definitional
• incongruity is too complex a word.
• what other words could replace testing in the definition and it still hold?