In the industry, a typical measure of quality is Defect Density, which is simply the number of defects divided by the units of work delivered (function points, LOC, or even test cases.) The other day, someone proposed that instead of (or at least in conjunction with) defect density, that organizations ought to measure first pass yield in testing. After a bit of discussion and thought about it, I don’t think this is a good measure.
- First pass yield (FPY) ignores vicious rework loops when test cases fail. A test that doesn’t pass on the first run falls into the other bucket, but no matter how many subsequent failed attempts and new bugs are created off that test case, it all is lost. FPY won’t measure that.
- FPY is redundant to defect density. Even if FPY didn’t have shortcomings, why measure the same thing twice?
- FPY doesn’t measure collateral damage. In software, it’s quite easy to fix one thing and then break another piece of functionality. So, a test that passed on round one of testing may fail on round two. FPY would say this test is good, when in fact, it is rendered bad by a related activity.
- FPY encourages the wrong behavior – to run test cases that you know will pass rather than to run high value test cases. If I were encouraged to get a high first pass yield, as a developer I would be guiding the test team as to what to test and when so that they never tested anything I thought was suspect. In the same vein, if QA was pushed (and it would be horrific if they were) to drive up FPY, they’d write worse test cases. The goal isn’t to have test cases pass, the goal is to fully appraise the system, and if it is of poor quality to report that.
No, I think we should stick with defect density and forgo FPY when it comes to testing.