Einstein once said that the definition of insanity is doing the same thing over and over and expecting different results. In software quality, running the same test over and over again, expecting to find a new defect matches that criteria. Far too often, developers perform “unit testing” that’s really integration testing. Then, the formal QA department performs more integration testing, which is probably really integration testing. Finally, the users perform “acceptance testing” which is again, integration testing.
We don’t really distinguish these various types of testing from one another or what the goals are, so instead, each group covers the same ground in turn, hoping to find something that the prior group didn’t. First off, it’s sad that they do. And the reason that happens is there are defects created by inconsistent build and software promotion processes and incomplete checks performed when the tests do occur.
But, even poorly run, the odds of finding defects this way isn’t great. Once fixed, except with the most disastrous of development teams, the bugs aren’t going to be coming back. So, instead, focus each team on doing a new type of testing so that new ground gets covered:
- Developers perform unit tests which check that the methods or classes are robust and behave well. They check for off-by-one errors, null pointer exceptions, etc. They might perform some integration tests, but not exhaustive ones.
- QA performs integration tests which makes sure large parts of the system hold together and deliver the right functionality.
- Users perform acceptance tests which makes sure the system performs as they expected it to. This is different from an integration test in that they’re more concerned with the end outcome than what might happen to the database or files in the interim steps. If code quality is good, acceptance tests can be performed in parallel with integration tests. If you find yourself having to wait until the end to perform acceptance testing because you don’t want the users “exposed to bad code.” then there’s something else entirely wrong with the way you develop software.
Testing should be a non-event, not the main event. You’re placing far too much risk at the end of the project if testing is the only way you find bugs, particularly if the way you test is the definition of insanity.