I’ve been thinking on the topic of how to decide where you should focus your energy when improving a process. In software, since the widgets you create are not the same size, we’re forced to use various factors to normalize for the differences just so we can make comparisons. Thus, the rise of lines of code or function point measures.
Once you start normalizing, then you can start talking about disproportionate risk. That is, what activities are far more likely to create a defect than others. Using this measure is helpful, except in certain circumstances.
Consider medical studies, for example. Almost always the effect of some undesirable behavior (smoking, for example) is expressed as an increased risk – “you double your risk of heart attack if you smoke” (I made that statistic up, by the way). The idea being, since you needn’t engage in smoking, reporting the disproportionate risk makes sense. If you just avoid doing X, then it’s less likely that Y will happen.
The problem is, in software (and manufacturing), we can’t not do the thing. We’re being paid to do the activity, such as create new software, so how risky it is rendered somewhat moot. We simply must seek to do it flawlessly.
The second issue is, that with the frequency of activities varying, you can end up with a lot of bugs from a relatively safe process. If you do something a million times, and it has a 1% failure rate, you’ll have 10,000 defects. However, if you do something 10 times and it has a 100% failure rate, you’ll have 10 defects. Which do you focus on? The second item has disproportionate risk compared to the first, but you also don’t do the second thing that much, so although it’s guaranteed to introduce a defect, it doesn’t happen that often. Instead, the thing you do all the time, but fails rarely becomes the frequent problems everyone sees.
That raises another issue. It’s easier to improve something that fails really frequently (lots of opportunity to improve) vs. something that fails rarely, even if you do it a lot.
In my opinion, disproportionate risk doesn’t matter all that much in software. Whatever is creating the most volume of defects is what matters to your customers, because its what they see all the time. You could make that process you only do 10 times a year perfect, but your opportunity to reduce defects is only 10… a drop in the bucket compared to the 10,000 you’ll generate from your more frequent process.
When all things are equal (both processes used about the same), then disproportionate risk matters. When one process happens far more frequently than the other, you have to focus on the one that outputs the most defects in aggregate, even if the relative risk is low. Yes, it’ll be harder to make improvements, but customers experience real failures, not understanding the odds of failure.