One of the most common testing tools is equivalence classes. Most testers generally understand what they are and how to use them. They allow us to do full coverage of an application without needing to test every possible permutation of input. Yet, despite their popularity I find that the two most important, the two most vital classes, get overlooked. Here I’ll cover these two classes and their implications on testing.
The Two Classes
When dissecting input into classes we usually think about in domain partitions like numeric ranges. Those are very common introductory examples since they are easy to explain. That is, if your application accepts a number between 0 and 10 you might have the classes: < 0, 1-5, 6-10, >10. This is a common break-down, but not necessarily the most logical starting point. Rather look at those two outer ranges compared to the inner ranges. The outer ones are invalid, and the inner ones are valid. These are the two most important test classes.
There are many good reasons to start at the level of valid and invalid. If you jump too quickly into further partitions you will likely prioritize your testing incorrectly. Consider that as a priority your application must function correctly with valid inputs — there is zero purpose to using an application that can’t work with valid inputs. Second priority is the invalid inputs, these are what determine the stability of the application. Beyond just prioritizing the manner in which the valid and invalid classes differs greatly. While both classes end up as part of the same test suite, the tactics used should be distinct.
The Valid Class
All inputs which the program is supposed to accept and process into meaningful results comprise the valid class. In terms of priority this class has to be the highest. That doesn’t mean it should get all of the resources, just the significant portion thereof. We arrive at this knowing that an application which can’t fulfill its promised features doesn’t provide value to the end user.
Inside the valid class you should do further partitioning. At a minimum you should test the boundary conditions (but only the valid ones) and all enumeration options. Once you start doing permutation of inputs it is best to stay within the valid class. That is, I have not see that mixing a few valid inputs with one invalid input offer significant value beyond just testing that one invalid input. Remaining within the valid class also greatly reduces the combinations allowing for a greater overall coverage of the application.
Tests for the valid class can be more strict than those in the invalid case. This is how the application is supposed to work, so it should work as described. Whereas users tend to be a little more accepting of error handling they simply don’t tolerate errors in functionality. Defects found from these tests are thus easier to triage as people can immediately understand the issue. This is an important distinction from the invalid class where you often get resistance to accepting what are seen as exotic issues.
Test case prioritization is a rather uncomplicated affair with the valid test class. There aren’t too many options here for leaving out any inputs without leaving the application’s functionality in doubt. With a clear separation from the invalid class you can also be certain that these tests get an appropriate share of the overall testing effort. And again, the easily accepted issues of this class will help justify extending the testing resources.
The Invalid Class
All inputs which the program is not expected to process comprise the invalid class. These types of inputs are generally tested to assert the stability of the application or to assess the user friendliness. Don’t take those further examples lightly either, unlike the valid class, tests in the invalid class should have a clear purpose. Answer the question, “What are you trying to determine with your test?”
The invalid class is vast in comparison to the valid class. You not only have semantic partitions but you also get to deal with invalid syntax, and out of domain inputs. Approaching this class is significantly different than approaching the valid class. I find it makes far more sense here to partition the expectations rather than the input itself.
For example, you have a simple input field in which the user should enter a value between 10 and 20. To test the invalid class you wish to test three possible effects: a semantic error, a syntactic error, and an overload recovery. For the first test you enter the value “5” and check if the program reports an error. For the second case you enter the value “abc” and check if the program reports an error. In the third case you submit the form with 200MB of data and ensure that it doesn’t crash.
Thinking in terms of effects for the invalid class allows you to construct input which is outside the domain of the initial input. In other words, if you are thinking just about numbers for an input field you won’t likely come up with “abc” as one of the equivalence classes. Nor will any partitioning algorithm ever tell you that a 200MB entry is a class amongst the integers. Tactics specific to the invalid class will help arrive at these types of tests.
Here again the prioritization comes up. You won’t likely need to do the 200MB test on all input fields, nor if you understand the UI framework will you need to perform the syntactic test every time. But you need to have a framework by which to judge that. Further to this I would argue that you don’t even need to test all invalid boundary conditions. Since all those boundaries are likely handled the same by the code you won’t elicit any more information by iterating through all the boundary conditions. Save the time for something else more important.
Conclusion
Valid and invalid are the two most important input classes for any input. This should be the first division made and further work within each class should be treated distinctly. By following this approach you will be able to better direct the entire test effort and avoid creating volumes of under appreciated defect reports. It may be a slight diversion from common practice, but the benefits are simply too great to ignore.
