How to construct a good ROI

Far too often the proposed cost benefit analysis that I see aren’t worth the paper they are printed on. For the most part, this stems from the benefit side of the calculation and not the cost side. Although we do make poor estimates, getting better at estimating isn’t terribly difficult. Start by reading Steve McConnell’s “Software Estimation” and you’ll be well on your way.

On the benefit side, this is where things go haywire. Lets say we’re talking about the benefit of better code reviews. There’s lots of industry data that indicates code reviews are valuable when done well.

So the math in people’s heads might go something like… Better code reviews reduce defects. Lets assume a test defect is … I don’t know … worth $1000 each, and that we can cut defects by 75% by doing better code reviews and that a code review can be done in ten minutes. Even if the basic formula is right, all the inputs are wrong. Just like a computer program, garbage in, garbage out.

In order to do the benefits half of the equation you need some data to help you with your assumptions. These things you assume are likely knowable, or at least we can get in the right ballpark. Want to know what it costs to fix a defect? Do a brief time study exercise. Or, if you know the cost of a production defect (which for some reason we seem to often know) then use research like Roger Pressman’s to arrive at an approximate cost of finding the defect in the testing or coding phases. The number is probably closer to $500.

Next, look at what the industry data has on efficacy of code reviews. A 65% improvement is not unheard of, but assuming you’ll capture the entire benefit plus more right out of the gate is pure optimism. First off all, you might be doing some reviews today, which blunts your benefit because the potential gain is smaller. Secondly, you won’t be able to capture the entire potential benefit most likely. In one example I looked at, the difference in defect density between teams that did and didn’t do code reviews was 20%. So, if effective code reviews are 65% effective, the maximum opportunity was only 40%, not the proposed 75%. Worse, when buying third party tools or services, you can’t rely on the sales person to provide you good numbers. They have a vested interest in you buying the product and thus in making the ROI work.

And then on the ongoing cost side, it takes a lot longer than ten minutes to do a code review. All in all, code reviews are certainly worth it, but you won’t get this too good to be true benefit from them. In many cases, we have a solution in mind, but no idea how much benefit we might receive so we make up numbers. Sure, that fills out the required paperwork, but it really isn’t due diligence. We have an obligation to support our assumptions with some data (our own or external).

Like reading one chapter of a novel

A friend of mine shared an excellent analogy about commenting code the other day. He referred to code without comments like reading one chapter of a novel and expecting to understand the whole story. It really doesn’t matter how well the chapter is written, there’s just more surrounding information which makes it make sense.

This stemmed from a conversation regarding commenting philosophy in code. I’ve recently heard a couple times that code comments are a weakness and represent a failure to write the code in a transparent manner. That’s like saying that the last chapter of The Great Gatsby is a failure because you can’t understand the whole story by reading just that chapter.

Code does not exist completely stand alone. It must interact with other parts of the system, and depending on the design may or may not do certain things. To understand what a function does may require understanding what the child functions do and don’t do and how the entire transaction hangs together. Particularly, when we have to call external services, it’s helpful to have context. Code comments are like the Cliff’s notes or margin notes in a book. They help you quickly tie back to other parts of the code and understand why the code is doing something, not necessarily what it is doing,

I wholeheartedly agree that writing “//increment i” just before the line “i++” is a useless comment, but there clearly are uses for comments. Comments are not just those things used by “weak” programmers who can’t write code clearly enough. They serve a valuable purpose, particularly in understanding the intent of the code during code reviews.

The cost of everything, but the value of nothing

There’s a programming joke that I heard a long time ago that went something like “LISP programmers know the value of everything but the cost of nothing.”  The point being that LISP was a very powerful language that had basically no capability to use system resources efficiently.  When I used to program and LISP, back in college, this certainly seemed to be true.

However, that’s not what I wanted to write about.  It’s just that the joke popped into mind for some reason, and it made me think about a project failure a long time ago.  About 10 years ago, I used to develop software for a Point-Of-Sale system.  Retailers are notoriously cheap, so while technology had advanced quite a bit, some of the systems in the field were still 386 based processors.  As a result, we had to cater to the lowest common denominator in the field, and for reasons I forget, this meant that we had a fairly slow build process.  It took about 2 hours to build our system to get a distribution ready that could be installed into a test environment.  And then, the install process took another half an hour or more, depending on whether you went to floppy disks, or built a distribution package via the deployment tool.  All in all, it was slow.

If you were a decent developer, you didn’t go through this process too much.  You did all the debugging at your desk and only did the full builds when you were ready to test in the lab environment.  For the most part, we could simulate pretty much everything, but it was helpful to go to the lab for some reasons that aren’t worth getting into.

At any rate, one day, we delivered a package to our QA department and it wouldn’t boot.  That was a very strange event, since we couldn’t recall at time when the entire system simply wouldn’t come up.  After some researching we figured out what had happened.  The build wasn’t good because it wasn’t complete.  And why wasn’t it complete… well, that comes back to the original joke, except in reverse – “the cost of everything, but the value of nothing.”

See, a developer knew full well what the cost of our build process was – 2 1/2 or more hours of time.  They were in a rush, so instead of running the full build, they grabbed the pieces they thought they needed from their desktop, assembled a package and threw it over the wall to QA.  Had it gone well, we’d be none the wiser it happened.  But it didn’t go well.  Because they had circumvented the build process, we ended up with an invalid build, and instead of losing 2 1/2 hours to the build process, we lost almost a full day trying to figure out what was wrong plus the confidence that QA lost in us.

Developers are a lazy lot – after all, we spend our entire lives automating repetitive tasks.  Saving 2 1/2 hours seems like a good deal.  After all, if we could take an easy quarter of a working day off our delivery timeline, it seems like a no-brainer, right?  Well, as it turned out… not right at all.

Process has value in that it prevents errors.  The build process is but one of many processes we partake in.  When all we see is the cost of something, instead of the value it provides, then not doing the activity makes a lot of sense.  Now, by all means, if the costs outweigh the value, that’s an entirely different story.

But as a point of reference, it’s about 25 times more expensive to fix a bug in test than during coding, and about 100 times more expensive to let that same bug to production.  So, when you think you’re saving cost by cutting back, consider the possibility that you’re driving up future costs in exchange for a small savings now.  What’s the value of that?

Standard deviation matters

I was recently reviewing two health plans which I could join.  One was a typical HMO-type plan and the other a high deductible plan.  High deductible plans have been pushed pretty heavily lately, what with the focus on rising health care costs, the recent legislation, etc.  I don’t want to jump into the fray on the political aspects, but I do want to share with you an observation on being risk adverse.

The way that a high deductible plan keeps premiums down is by requiring the policy holder to pay everything out of pocket until you reach some limit.  For that reason, high deductible plans are often called catastrophic coverage plans, since they only become helpful if you incur massive costs.  In exchange, you pay very low premiums.  And that’s because you get very little benefit most of the time.  These plans, from everything I’ve read online, are great for young and healthy people, but not so good (obviously) for people with chronic conditions or families.

I am married and have two kids, but I was willing to run the numbers and look at the odds to see whether I should go with a traditional plan or a high deductible plan.  First you have the premiums to pay.  If you use no care than a high deductible plan beats a traditional plan hands down in premiums.  The rest is an odds game.  How many times will you have to visit the doctor?  How many prescriptions will you need?  What are your odds of having a catastrophic illness?  For a traditional plan, each one of these events costs you just a copay – typically $20.  For a high deductible plan, you have to pay the full amount until you reach the yearly limit.  (Never mind the fact that if you have a chronic condition you’ll reach that limit year after year after year).

Because I’m a data person, I happen to have tons of data on how often we visit the doctor and my current insurer was kind enough to send me a complete history of all charges so I had a reasonable idea of what various things would cost me.  Then, I built two models, one using a traditional plan and the other using a high deductible plan.

Guess what, in the end, odds were that even with my family (and we make our fair share of doctor visits), I’d likely pay less with the high deductible plan than with a traditional plan.  It wasn’t true in every situation, the potential cost to me of the high deductible plan overlaps with the potential costs of the traditional plan, but the premiums for a traditional plan are so much higher that it was difficult to overcome them.

But, there was something else I observed.  Not only did my models calculate the average result, but they also calculated the range of possible results.  And not surprisingly, the high deductible plan had a much, much higher standard deviation.  This isn’t surprising, since each medical event in a high deductible plan costs a lot more, and so the variability in the result is potentially much greater.

Ultimately, that’s why I stuck with a traditional plan.  Yes, I’m very likely to pay more, but the result is predictable, and I like predictable outcomes.  (I also like not having to concern myself with whether its worthwhile to go to the doctor).  If you’re a risk adverse person, and if studies about investing are to be believed you are more risk adverse than you think you are, then a traditional plan produces a more comfortable financial result.  It’s like buying bonds instead of stocks.  You may not come out way ahead, but it’s much more unlikely that you’ll come out far behind.

Consider that when designing a process as well.  Consistency is important.  You can end up with a software development process that sometimes produces perfect code, but sometimes produces awful code, or you could have a process which produces consistently average code.  Which one is easier to deal with for continuous improvement?