I was just out to lunch with a friend who was telling me that the quality assurance department they work for was switching over to risk based testing. It’s a simple concept as I understand it – test more where there is more risk, test less where there is less risk. Risk is determined via experience, typically in the form of some scoring system which rates how risky a given change or application is. The higher the score, the more or different types of testing you do.
Now I’m not a testing expert by any means, but the conversation turned to how they were going to measure their success. Prior to risk based testing, the measure of success for the department was defect containment rate (DCR).
Defect containment rate is fairly basic as well. It’s simply (every defect you find in testing) / (every defect you find in testing + every defect you find in production). In effect, if you find 75 defects while testing and after the code reaches production 25 more defects are found then you have a 75% (75 / (75 + 25)) defect containment rate. Generally, the higher your DCR the better.
But, no, I’m told by my friend that the new measurement will be on defects found for the areas QA tested. So, by that logic, if through risk based testing you determined function A wasn’t risky enough to be worth testing, and it breaks in production, that defect shouldn’t be counted against you… Such a decision would only serve to affect the denominator. You’d still report all the bugs you found in test, but for each prod defect that was found, you’d get to decide whether or not you meant to test for that bug. Suddenly, 25 defects in production might only count as 10 or 15 if you deemed the remainder as “things we weren’t looking for.” Now instead of 75% containment (75 / (75 + 25)) you’d have a 88% (75 / (75+ 10)) containment rate. Hey, you improved!!! Wrong!
Something is amiss here! Since when did just because you changed the way you do things change what is important to your customer? If your prior measurment – defect containment rate – measured what your customer expected of you, where’d you get the free pass to not accomplish that goal anymore?
You don’t design metrics around what will look good. Looking good is NOT equivalent to doing good. Actually being good, and meeting your customers’ needs is the goal. If you refuse to measure how defects impact your customer just because you weren’t looking for those defects, it doesn’t make the defects go away.