Cost Per Defect – not so economically crazy

Capers’ Jones, one of my favorite authors on metrics in IT, has often called “Cost Per Defect” a bad idea.  His argument is that in situations of high quality, cost per defect will be higher.  And one can see how this math is true.

If you spend $100 (1 hour @ $100/hour, let’s say) testing something and find 1 bug, the cost per defect is $100.  If you spend $100 testing and find 10 bugs, then the cost per defect is $10.  In this case, it’s cheaper to test where quality is poor.

Now, I am not advocating a test-only approach because it makes cost per defect cheaper.  But I think this mathematical reality raises a much more important question: if quality is good, why do you continue to test?

Testing may very well be a fixed cost regardless of quality.  You, in theory, don’t know if the code delivered to you is good or bad.  Therefore, you write a set of tests assuming that it all needs to be covered and run those tests.  If the quality is good, it appears expensive to test and if the quality is bad, then testing appears cheap.  Thus, Jones argues, cost per defect encourages poor quality.

I argue it encourages something else – an opportunity to revisit whether you should be testing at all.  Jones argues that the cost per defect measure is economically perverse because it punishes high quality by making testing look expensive.  I’d argue that testing was never value added in the first place.  Therefore, when high quality is delivered, why do you continue to waste your time testing?

Instead, I propose that cost per defect is a great measurement for a test organization.  It clearly articulates the issue that testing isn’t of value where quality is high.  And therefore the cost per defect is high.  When no defects are found, cost per defect is undefined, and although this is mathematically annoying, it represents a measure of pure waste.  Testing and finding no defects has absolutely no value to your customers.

If you think testing in quality is the way to achieve good code, then Capers’ argument makes sense.  If you’re always going to test regardless of the situation, then it is an odd measurement.  If, however, you think that testing should not be conducted (or at least done far less) when you know the quality is good, cost per defect drives the right behavior – less testing where quality is good.

2 thoughts on “Cost Per Defect – not so economically crazy”

  1. It is easy for people to grasp if they think about expensive cars or appliances that they use in their normal lives. Is a Honda cheaper to fix than a Kia? (Just examples.) Which is higher quality? Or is an Acer pc cheaper to fix than a Mac Airbook? (Again, just examples.) Probably, but which is better made to begin with?

    The idea isn’t that alien once people recognize that it exists in all manufacturing.

    1. Interesting point. In another blog entry I wrote a long time ago, I bemoaned the “you get what you pay for” mentality. In that particular case, I had just gotten Vonage as phone service and was having some trouble with reliable connections, but it was costing me less than half what my traditional phone company was costing. It’s been years since then, and I’m still with Vonage, because I eventually figured out the handful of tricks that make it far more reliable on my network. Suddenly, I’m getting far more than I paid for – phone service (that appears to me) as good and reliable as traditional service, at a far lower cost for the features.

      But, the cliche exists for a reason. You do tend to pay more to get higher quality up front, and in the long run it should be worth it. Though there certainly are extreme luxury items which are expensive solely for the purpose of being expensive and not particularly better than a high quality mainstream item that serves a similar purpose.

      As you pointed out, you can either put the money in up front OR pay less up front to get an apparent “cheap cost to fix.” My ideal is to break this paradigm and solve both – deliver a great quality product that doesn’t cost very much, and that’s where my LEAN bent comes from.

Leave a Reply

Your email address will not be published. Required fields are marked *