Systematic testing is ordered, planned, and testing designed to be purposefully methodical in its approach. This contrasts wtih ad hoc testing in which testing is executed with a special purpose (although some people mistakenly assume ad hoc to infer arbitrary actions or silly randomness). Systematic testing also contrasts with exploratory testing which is useful in initial examinations, where testing time is relatively brief, or when learning general software behavior.Systematic testing requires an in-depth analysis of the application and the application's components at a very granular level. Some systematic testing approaches include equivalence class partitioning, boundary value analysis, combinatorial analysis, state transition testing, basis path testing, etc.
After years of working with testers. I have come to the conclusion that they are not only a very destructive lot of people, they tend to be relentless when it comes to exposing issues. These testers are like a dog with a bone. Unfortunately this scenario typically plays out during the execution phase of testing, which in most cases is way too late.That brings me to my next epiphany: test early and often. If you can test early and often during the application lifecycle development process, not only will more defects be found, it will also be easier and cheaper to fix defects if they are found earlier in the process.If you’re constantly executing tests you can’t spend your time properly documenting test cases or completing risk analysis unless you can move objects with your mind? While at first I was very resistant to the concept of exploratory testing; I did see definite value add when it came more testing and less prep work. Exploratory testing doesn’t bog testers down by having them constantly document possible scenarios and use cases—which only a few are converted into the regression tests set. But without structure comes chaos and one thing most testers don’t like is confusion. Confusion leads to the dark side of the force.
Exploratory testing in a nutshell takes advantage of the experience level of a manual tester on a specific application and allows them some autonomy when it comes to deciding how and when to test new functionality. It also gives them some independence when it comes to regression testing, also known as shotgun testing, because you cover a lot more area. This method leaves bigger gaps in your testing of an application; however, sometimes this can be too big a risk to the overall performance of the application.Once you find an issue or defect, can you repeat those same exact undocumented steps time and again? A lot of times the tester will find an issue during exploratory testing and to correctly document the issue he or she must duplicate that same issue two to three times before handing the issue over for resolution which sometimes can be frustrating and monotonous if the issue is intermittent.Before getting into much hot water with exploratory testing methodology advocates the theory is that a person should document test steps as they go. The reality is that documenting is very time-consuming and documenting your steps is only practical if a defect is found or it has been decided that this test case is a candidate for regression testing.Depending on your experience level with the application and the amount of training on the methods of exploratory testing can really pay off. Like Anakin, too much emphasis on one person’s abilities to foretell application’s behavior in production place a lot of responsibility on a single person shoulders and can lead a project team to have a false sense security? This scenario plays out quite a bit when using these lean development methodologies. For example You pay a testing 60k a year on a five million development project that could generate tens of millions for the company.
To answer the original question, when faced with a Spock (systematic approach to testing) versus Anakin (exploratory approach to testing) situation (without having the benefit tools and the constraints that sometimes imposed by different development strategies) I would always choose the methodical logical systematic type of testing. But I am not done with my train of thought here. With the development of tools like Sprinter and the introduction lean development strategies exploratory testing seems very appealing at times.There is no one de facto statement I can use that could clearly define one strategy over another. For example, I believe if systematic testing is structured correctly it can work seamlessly in an Agile environment. If you have methodically driven team of testers, the use of exploratory testing on a critical or large development effort,( taking into account a groups of testers experience level) could speed up your testing endeavor and still lower your overall risk to the current initiative. But in most cases there are never just two criteria that you can base your decision on.When your team is selecting a testing strategy your group must weigh a myriad of information to make the best choice for your team. But choose wisely young padawan, and remember—Vulcans never bluff.If you find yourself in this dilemma please feel free to reach out and comment on this blog with your criteria sets. Please include your preferred testing methodology and we can work with you to mitigate some of those issues so you can make a more informed decision.
