Since it’s the World Cup right now, it makes sense to spend some time talking about keeping score. I’m from the US, so I took note of the US beating Ghana 2 goals to 1 recently. That’s the great thing about keeping score, you actually know who came out on top.
We use scorecards (or something equivalent) widely in professional sports. In baseball, for example, each player has a set of statistics associated with their performance – batting average, ERA, RBI, and so on. And using these statistics we have some clue as to which players are better than others. That doesn’t mean we can exactly discern two great players, but we can get them into a general stratification and figure out who to keep and who to cut.
In software we often desire to have scorecards, but we fail to use them appropriately. Perhaps we consider three things important about every project – on time, on budget, and with decent quality. Sure, it’s a simple model, but perhaps better than nothing.
So we go and collect this data on projects and we notice one project who, based on our data appears to be late, over budget and below average quality. So we go ask the team what’s going on… And wouldn’t you know it, they’ve got a reason for everything. The scope changed, so they can’t be held to the date, and they couldn’t get the resources they really needed for the project, so people are working overtime, and well, you know, if you don’t have top people then the quality will suffer. Clearly this project’s scorecard says it is in trouble, but the team wants to call it all clear.
The outcome of software projects is often like the outcome of a sporting event. The project will either eventually succeed or fail, just as will one of the teams. Some times, for both groups, the getting there isn’t pretty. In the end, the winning sports team may say things like “we won, but we could’ve played better” which demonstrates a far better understanding of probability of success than most project teams have. Instead, the project team wants to put out its own version of reality, one that is inconsistent with how all other projects are looked at. Suddenly you don’t have a scorecard anymore. Who won the game when the score reported is 2 to Purple-People-Eater? You haven’t a clue.
I’m not proposing that measurement in software is an easy task, but if
we choose measures that act as a proxy for a project or organization’s performance, we must use a consistent ruler to measure them all. Otherwise, you don’t have a scorecard, you just have, as Kaiser Fung likes to call it, story time.