This one graphic DOESN’T explain it all

How many times have you been reading Facebook, or your favorite blog or a site like buzzfeed and you see an entry with the title like ‘this one simple graphic explains [insert topic here] once and for all’ or something like that.

These titles suggest to the reader that if you just looked at some problem in a specific way that suddenly it’d all become clear. Of course, the next step is that your democrat or republican friends post these items to Facebook with a helpful comment like “for my [opposing party] friends.” And really, nobody’s mind is changed.

First off, I’m not going to spend much time addressing cognitive dissonance. Reality is, giving someone evidence that isn’t in line with their world views tends to lead to a strengthening of their current views, not a weakening of it.

But secondarily, for any sufficiently complicated topic (which in my world, is pretty much all of them), there is no one graphic that explains it all. And I suspect most of your situations are like that as well. Let me use an example, organizational productivity. We measure effort per function point per industry norms. And we were demonstrating in our “one chart that explains it all” that productivity had improved since a recent change. Except one chart won’t do it. The chart makes the main point, but then we had at least five other charts checking things like the measurement system hadn’t been tampered with, that quality hadn’t suffered as apparent productivity rose, and so on. In IT, most of the things we measure are proxy measures for some outcome we care about. As proxy measures, we always have to worry about the measurement system and unintended consequences of our choices. As a result, no analysis is ever complete on a single chart.

Treat anyone and anything that is explained to you in “one simple chart” with suspicion. If it seems too simple and obvious, it probably is.

The “right” way for the business, but the “wrong” way for us

How many times have you been annoyed by a business partner who tells you exactly how something should be done. Let’s say you’re building a new application and it needs data from a third party. The third party provider has a web service you can call to get a realtime result. However, another application in your organization is already getting that data periodically as well. The business person begins to question your design (and the associated cost of a new call to the webservice) when wouldn’t it be better just to take a nightly feed from the other system and load it? For oh so many reasons, we know this isn’t the right thing to do. The data is likely outdated since it’s now 24 hours behind the realtime version. It ties two unrelated systems, who happen to need similar data, together. Now if the old system changes, you’ll have to modify and retest this new system as well.

If, as developers, we were purists about doing the right thing, it’d be a defensible stance. But how often have we kludged something together ourselves? Take for instance, data we need about our organization. It might come from a Project-and-Portfolio Management tool. Perhaps we want to look at data by “team,” but for whatever reason the team data isn’t available in the tool. Do we try and do the right thing and fix it? Not usually. Just the other day I encountered a data extract, then joined via Microsoft Excel and a vlookup() to a list of people which mapped them to a team. Why was the team data not in the tool? That would’ve been difficult, but the right thing to do. Now, instead of a clear single system of record for team membership, both the PPM tool’s org structure and this one-off would have to be maintained. And while we are often hesitant to make compromises about the stuff we build for the real customers, we will accept horrid messes that create work for ourselves. Why should we be unwilling to have messy solutions imposed on us but willing to impose them on ourselves?

I don’t think we can hold others to higher standards than we are willing to hold ourselves as developers. Don’t create your own messes.

Following fast

Some time ago, unfortunately I can’t cite the source, I read that one of Toyota’s major strategies was not necessarily to innovate but to be able to follow quickly in the footsteps of innovators. The point was, whether or not correctly attributed, innovation is overrated if you can copy someone else’s innovation quickly enough.

Well today I saw this article on Huffington Post… A sort of rant about why 2048 was a rip off of “threes!” and how you should go out and buy Threes! because they put effort into their game and the ripoff only spent a couple weeks being constructed.

I won’t comment on the justness component of the article, but I do think that it’s an excellent illustration of following fast. 2048 is indeed very similar to its predecessor. I’ve played both since my father-in-law first challenged me to beat his top score on Threes! and subsequently pointed me to the equally addicting and free game, 2048. However, 2048 does have some differences and for the casual game player it’s far less frustrating. So, should every company invest 14 months inventing a game, or is a two week knockoff done by a single person and given away adequate?

Particularly in game playing, the marketplace is a highly commoditized place. If you need an entertainment fix, almost any game will do, so while each game is unique, you’re not just competing against other games like yours, you’re competing against all other games which satisfy the same need.

For better or worse the free market is pretty indifferent towards fairness, so recognizing that and following fast may be the way to go. And oh, it has another advantage, one that was articulated by Tom Demarco in his work “The Deadline.” If you have an existing product to copy, specifications for how your product should at least mostly work are right there in the form of the existing software and user manuals.
>

My “free market” sanity test

Every time someone tells me about the hottest new thing in IT, or whatever for that matter, I simply apply what I call my “free market test” to deciding whether there’s anything to it.

I like to pick on Agile, as you well know, so let’s use it for a comparison. We will compare it to outsourcing/offshoring. Now, back in 2001 when the Manifesto was signed, IT outsourcing wasn’t a new idea. I can’t pin down a date easily, but it looks like the ’90s saw a significant shift in IT outsourcing coincident with the rise of the internet. (If fairness, I could’ve easily picked on components of Agile as well. Most of the ideas in Agile are not new. Consider the small team approach. It can be traced back at least as far as Brooks’ seminal work “The Mythical Man Month” in his surgical teams approach.)

Here’s the thing. 10-15 years on, outsourcing is huge business. I haven’t dealt with a company who isn’t doing it. Agile is mixed. If the hype is to be believed, it should be the savior of all IT. So why is outsourcing ubiquitous and Agile is not? I think the free market test answers that. If something works, it will get widely adopted. No sane company is going to allow their competitors an advantage that they can easily copy. And any company can easily copy IT outsourcing or Agile.

The free market test is a simple way to think about whatever “new to your company” idea that may be going around. If everyone is doing it, chances are it is working. If it didn’t work, companies would abandon it, or those that continued to pursue the path would go out of business. The free market, like Darwin’s evolution, is survival of the fittest. Behaviors that don’t markedly improve fitness die off. Behaviors that do, thrive.

What can a snowblower tell us about software?

If you’re in the north eastern United States, you’re probably thinking about snow right now. And if you’re responsible for clearing the snow from your drive or walkways you might also be all too familiar with time behind a snow blower. For years I hand shoveled my walkways, but when we moved to this new house they were simply far too long for that.

It takes me about an hour to do all the clearing I am responsible for, so that’s a lot of time to think, which isn’t necessarily a bad thing. This particular snow is the deepest we’ve had yet. My snowblower has three forward speeds on it and presumably you use a slower speed when you have more snow to clear. The slower speed allows the auger to clear the snow before it gets all backed up.

So, as I was clearing the drive, I noticed something. Even at the lowest speed there was enough snow that some of it was being simply pushed out of the way by the blower. That meant that I’d have to do clean-up passes just to get that little bit of snow that the blower wouldn’t get on the first pass. And that got me to thinking. What if I just went faster? After all, if I was going to have to make a second pass anyway, who cares if it’s a tiny bit of snow or a little bit more?

And that got me to thinking about software. One approach might be to take it slow and carefully, but if you’re going to create bugs anyway, then perhaps going that slow isn’t the right answer. You’re still going to need the clean-up pass so you might as well let it happen and just clean up a bit more, right?

That sort of makes sense, if you think a second pass over the code is as effective as a second pass with the snow blower. In terms of dealing with snow, the blower is relentless. If it goes over the same ground twice it will do so with the same vigor as before. On the other hand, testing is imperfect. Each pass only gets about 35-50% of the defects (tending towards the 35% end). It isn’t like a snow blower at all. If you push aside a relatively big pile of snow with the a snow blower, it’ll get it on the second go. If you create a big pile of bugs in the code on your first go, a round of testing will likely reduce the pile by less than half. Then you need another pass, and another just to get to an industry average 85%.

There’s one other thing about going too fast that I learned with my snow blower. Some times you get a catastrophic failure. In my case, going too fast with the snow blower broke the shear pin on the auger. It’s a safety feature to prevent damage to the engine but it does make it impossible to keep using it to move snow. And software is a bit like that too. Go too fast and you may introduce a major issue that you’ll spend a whole lot of time cleaning up. It is not all about speed.

Want something fixed? Give it to someone who doesn’t want the job

I recently had a great experience with simplifying IT processes. Due to recent org changes, a home grown product (that was a ton of spaghetti code) got turned over to a new team. The thing was, the new team’s job wasn’t a support role and they didn’t particularly relish coding. Over time, the organization had become dependent on the product, although people suspected it was partially because they didn’t know any better. There were off the shelf tools which could do the same job.

Well, it turns out if you want to get rid of some job, the best people to give it to might be the people who don’t want the job in the first place. If you’re content building and maintaining a bunch of spaghetti code, and I then give to you an additional product, you’re likely to keep maintaining that one too. But, if you don’t particularly like coding, you are going to try and avoid doing it. One of the best ways to avoid doing something is to get rid of it.

In fact, this is exactly what happened. The team, who didn’t desire to fix everyone’s problems figured out how to replace the home grown product in just a few months. For years the organization had been told that it couldn’t be replaced. The difference? The old team was content to maintain it, perhaps even proud. The new team had nothing invested in it and didn’t want to.

“I don’t think necessity is the mother of invention. Invention . . . arises directly from idleness, possibly also from laziness. To save oneself trouble.”
― Agatha Christie, An Autobiography

Seeing through messes rather than cleaning them up

At the Cutter conference I recently attended, a gentleman named Robert Scott gave the keynote address. In it he talked about what he thought the future of technology leaders would look like. One thing he said stuck with me – future leaders will need to be able to see through messes rather than clean them up.

Although I found it an interesting idea, it couldn’t quite fit it into something I could relate to. That was until a discussion today. I was talking with someone regarding measuring a quality assurance organization and each time I proposed a metric he would counter with something like “but if we measure it that way then what about the situation where X doesn’t apply?” essentially implying that there was noise in the measurement system and we’d have to clean it up before the measurement would be useful.

And that’s when Robert’s comment finally made sense to me. As a data person, I’ve learned to live and even embrace the noise inherent in measuring software development. It’s more like economics or sociology than it is like physics or chemistry. Given some exact set of inputs, the output will be directionally the same, but not exactly the same. But software people don’t go for mostly right or directionally right. It bothers them. After all, software is either wrong or it’s right. Either it’s a bug or it isn’t. There’s no such thing as that’s sort of a defect. So, when you introduce a measurement system that naturally contains noise, the discomfort sets in.

Here’s the thing. If we assume Robert Scott is right, that future leaders must be able to see through messes rather than clean them up, then the measurement of software stands a chance of moving forward. It will always be imperfect, but the data helps us to make better decisions. Yes, it’s messy. It contains noise, but rather than always trying to clean it up, acknowledge it is there, and start to look beyond the noise at the signal underlying it.

Tom Demarco talks ethics

I never took ethics in college. I also never planned on attending a conference to hear a talk on ethics. After all, ethics were sort of a base assumption from my perspective and not given much thought beyond large companies having employees sign a code of ethics on some sort of frequency.

In fact, I thought I was attending a talk on decision making, and thus expecting something about decision theory, game theory, maybe psychology. Certainly not ethics. But Tom took something that at first I was like “where is he going with this” to “wow!” I’ll attempt to do it justice, but I can’t promise that I will. To the best of my ability, here’s what I took away:

Aristotle believed that ethics were logically provable. Metaphysics contains all the things we can know about the world. Epistemology is built on that. It’s everything we can derive from what we know about the world. For example, Socrates is a man. All men are mortal, therefore Socrates is mortal. Ethics, Aristotle “promised” were provable with logic. For something like 2400 years, all kinds of philosophers tried to make good on this promise and were unable to. At some point in time, David Hume perhaps, classified metaphysics as “is” statements and ethics as “ought” statements. It was argued that it was impossible to derive an ought statement from an is statement.

Along comes Alasdair Macinytre. He argues that if something is defined by its purpose (this is a watch, for example) then the ought statements follow naturally. What ought a watch do? It should tell good time. So, that raises the question, what is the purpose of man?

We go back to Aristotle. Aristotle also created a mechanism for defining things. His definition requires that you group something and then differentiate it. So, a definition for man might be “an animal with the ability to speak.” That’s an is statement, for sure, but by Macintyre’s requirements, it doesn’t define man’s purpose. Macintyre goes on to define man as an animal that practices. Creating practices becomes man’s purpose. A practice is something we attempt to systematically extend. Tennis, for example, is a practice. The current tennis players have extended the sport in new and interesting ways, such that although famous tennis players of yore would recognize the sport they probably couldn’t compete anymore (even if they were young enough) because the sport has been systematically extended.

So, if that’s right, that man is an animal who practices, then for each practice we create, the oughts follow naturally. If you are a software developer and your manager comes to you and says “we need to take three months off this project” what ought you do? Well, first you ought to be honest – cutting the timeline will hurt the quality of my practice. Second, you must be just – quality matters to our customer, and we can’t deliver poor quality. It’s a disservice. And lastly, you must be courageous – go back in there and tell them no!

How many times has one of our employees, by this framework, acted ethically and we viewed it as a problem? Far too many times, I’d guess. The person with ethics, who values his or her practice and whose ought statements are clear can be frustrating. But when viewed through the lens of Tom Demarco’s talk, suddenly what they’re doing makes a lot of sense.

Do something counterintuitive

A post over at one of my favorite management blogs reminds me of my own recent experience with going for it on fourth down. Recently I’ve been working on a project to improve estimating. It’s not uncommon to hear that estimates should be created by those doing the work. Indeed, if a random person unfamiliar with the ins and outs of your system (namely management) estimates a project for you, odds are it’s going to be bad. But we can take it one step further, what if there was evidence that even if the person doing the work makes an estimate you should override that decision based on a model instead?

Steve McConnell notes in his book on estimating that various experiments have shown developers to be eternal optimists. One way he argues to correct for this is to simply make estimates larger. Unfortunately, when evidence shows you have a bias, then you aren’t going to make the right decision on fourth down, so to speak. In our own research, a model helped to compensate for human fallibility. Although we still got an estimate from the developer, when we combined their data with historical information into a model, we got an outcome that outperformed expert judgement alone 65-80% of the time. That’s not perfect, but it’s surely better than without any model at all.

We always want to believe in the greatness of the human mind to make decisions and in a massive number of cases we don’t know a better system, but as Curious Cat points out, sometimes the evidence isn’t what you’d expect it to be at all.

What is our obligation as a data scientist in presenting information?

I just spent three great days at a conference. I hadn’t been to one recently, and a lot of times that I’ve gone I’ve been disappointed by the heavy vendor focus on selling products and giving away trinkets. This particular conference was much more intimate, and well worth my time to pick other people’s brains.

In one session about fifteen of us discussed designing effective dashboards. Although I love Edward Tufte, the conversation here never once touched on data to ink ratio or any of his other great ideas. Instead, we spent a large portion of the roundtable le debating the obligation of a data scientist to guide the dashboard design process.

For example, not that long ago Stephen Few held a dashboard design competition ( here’s his result). The challenge I had with this competition wasn’t in how well the data was presented. Indeed I learned a great lot by looking at the winning solution and Stephen’s solution. What left me feeling unsatisfied was the missing opportunity to discuss what should be presented versus how it should be presented.

And this, to me, is the central question of information display. There are many valuable rules about presenting information once you decide to present it but scant little advice on how to decide whether to present it at all. Statistics literacy is weak in many organizations, so as a likely stats expert in your organization, what do you owe them?

  • Help them to identify the outcomes that matter first. Absent this, no dashboard can be useful. It will just be a lot of beautifully presented garbage.
  • Help them determine potential leading indicators and help them assess whether they matter. It’s not enough to have good ideas about what might matter. We have an obligation to test the relationships.
  • Help them think about the sometimes subtle and insidious problems of statistics… False causation, mathematical quirks that change apparently relationships (like log transforms on a single axis or sharing a variable between two composite measures), and other things that will mislead.

If we fail to do these things for our organizations then we do a disservice to our science and to those that we work with. Dashboards should not be created simply to provide confirming evidence for the world view we want to hold, but to help us seek out information that disconfirms our beliefs also.