Cargo cults in IT

The end of WW II gave rise to a striking example of a causality problem. On small, remote islands, indigenous people encountered large numbers of military servicemen for the first time. As we, and others, landed ships on these islands, cleared jungles for runways, erected control towers and ran military drills the indigenous individuals observed newfound well being in the cargo our servicemen brought with them.

When the war ended, the military packed up their temporary bases and took their cargo planes with them. And with it went the newfound wealth of the native people. So, what did they do?

Well, they replicated the actions they saw taking place. They cut airstrips out of the jungle. They erected towers similar to the landing towers they saw. They executed military style drills. They carved wooden headsets to wear like the ones they saw servicemen wearing.

Since they didn’t understand the causality of the entire event – a war that led to need for new bases that led to airfields and cargo planes, they figured if they recreated the parts they observed they’d get the same outcome. Cargo planes ought to land and bring cargo. Of course, it doesn’t work that way, which is what makes the idea of a ‘Cargo Cult’ so interesting. We clearly see the logical problem in that situation.

When it comes to the corporate environment, however, we often fail to see our own Cargo Cult behaviors. We observe a great company, say Google, and we see that they have bicycles on their campus, so we buy bicycles for our campus. Or Facebook engages in hackathons so we start to do so too. But buying bicycles or performing hackathons is not going to make your company like those other companies. You are simply emulating behaviors that look like the other company without understanding the underlying culture that causes them to do these things. And as a result, you’re likely to get disappointing results from the mimicry. These companies aren’t successful because they engage in these behaviors. The companies are successful first and therefore may engage in these behaviors.

Which brings me took another point. We often look at older companies that fail and we observe that they don’t engage in these behaviors, and we use that as evidence that these behaviors are necessary to survive. But we can learn something from Mark Buchanan’s ‘The Social Atom’ on this point. In his book he demonstrates a ridiculously simple model that predicts the rise and fall of organizations, simply based on time and getting too big. You don’t need to model any behavioral stuff to get the effect as I recall. So, probabilistically, large old companies will decline even if behaviors are held steady. There will always be companies coming and going, and we will always be able to be selective and say “see, that company didn’t do X and they failed. We need to do X.”

If you find yourself saying that in the future, just remember that you may now be a card carrying member of a cargo cult.

Why change if nothing’s going to get better?

I’ve often wondered why we undertake some projects at all.  Often times we’ll upgrade a system or replace it solely for the purpose of getting to a “common infrastructure” or “common process.”  Now, not that there’s anything wrong with achieving commonality, but it has to serve a purpose greater than simply having something be common.

If someone said to you, we’d like all of you to drive Ford Fiesta’s (no insult intended, I just picked a car) simply because we thought everyone should have a common car, would you be on board with that?  No, of course not.  A Ford Fiesta might not meet your needs.  If you’re already driving a Toyota Prius, it might not even make anything better – assuming the only category for evaluation was fuel efficiency.

This was something I struggled with when taking my LEAN training.  On one hand the trainer indicated that you should never standardize before you optimize the process – in other words, only change if you are going to change for the better.  In another place the trainer indicated that without standards you cannot optimize.  So, suddenly you have a problem – if you don’t standardize (and therefore necessarily change) then you can’t optimize, but you shouldn’t change unless it is optimized???  Which is it?

I believe the answer is one of intent.  You may change/standardize if the intention is to then improve, but if the intention is to reach a standard simply for the sake of being standard, don’t do it.  What’s the point of change if nothing is going to get better?  Why add disruption if you don’t intend to make something better in the long run?  It seems silly.

A delightful discovery

Generally, I think it’s kind of silly to spend time patting yourself on the back about what you did.  Especially when there is so much more yet to be done, but I wanted to share an experience and an anecdotal measure of what a good process looks like.  (Apologies up front for the self-congratulating nature of this).

Almost 10 years ago, before I learned a thing about LEAN or Six Sigma, I worked for a large company.  When I was there, I put into place a very simple multi-stream waterfall process.  I didn’t know anything about LEAN, but I knew at the time that I didn’t want a burdensome process, just an effective one.  Books like Steve McConnell’s Rapid Development and my past experiences were my only guides.

I had no idea at the time if the process was a good one.  I had no idea when I left if the people who stayed would still use the process or abandon it the minute I was out the door.

Recently, I had the need to recreate something like that process I put in place 10 years ago.  You’ll find from my writing that I have not jumped on the Agile bandwagon.  Based on my own research, the work of individuals like Boehm and Turner in Balancing Agility and Discipline and a healthy amount of valid criticism from the Internet community, I’m convinced turning to a craft-like approach to software is not the answer.  Thus, my need for a simple multi-stream waterfall process…

So, I called up a friend who still worked at my former employer and asked for the old process documentation I had created.  I was convinced that they were no longer following it, so I should be getting back exactly what I had left them with.  To my surprise I was told that no they did not have what I wanted.  Not because they had promptly filed it in the trash, but because it had evolved!

They sent along a package of templates and flows which, while it strongly resembled what I had left, they had expanded upon and improved to meet changing situations.  The spirit of what I had created 10 years ago was still intact!

Two things are important about that experience.  One, I was with that company almost 5 years before I left.  That duration, the authors of LEAN Thinking point out, is about the amount of time needed to make a change permanent.  Had I left a year or two into the process change, it may have not survived.  For five years I pushed hard on my team for change, and I had left thinking that it was all for naught.  Not so!  The type of persistence that Womack and Jones write about paid off.

The other was that processes will continue to evolve.  On one hand, I was kind of hoping to get back exactly what I had left them with.  I was a little disappointed that I did not because selfishly, it was no longer quite mine.  But, after a moment of reflection, I realized that the fact that it was living and evolving was necessary to its survival as a viable process.  It also meant that team, many of whom are still there from when I was there, has taken ownership of it and made it their own.

It’s personally gratifying to see a process survive for so long and grow and change and be perfected but not abandoned for the latest fad.  But what’s more important, the team has been able to steadily improve by working from standard process and continuously, patiently modify it in little ways.  It’s heartening to know that Kaizen works.  We talk about it a lot, but how often do we get to look back 10 years and see its effects?

Quick and crude change

When I was learning about LEAN, my LEAN Sensei taught that changes made to the process ought to be quick and crude, as opposed to slow and elegant. It wasn’t until I was struggling with making a change recently that I realized why this is so important.

Unit Testing is a well understood concept within the larger industry, but it still amazes me how many organizations that I run into that confuse integration testing with unit testing or fail to do anything that one might even mistake for unit testing at all. In this particular organization, resistance to change was extremely high. A long, long history of believing that their particular coding environment made unit testing impossible had predisposed the team to not try it at all.

It turns out that tools actually exist to unit test in that environment, but I didn’t know this, and I was assured by the team that no such thing existed. As an aside, we often defer to the teams as being best positioned to make change, but I question the validity of that philosophy when they are blinded by what is, rather than what could be.

Setting off with the belief that no tools existed, I devised a crude way to perform unit testing in their environment. We tested it out with one helpful developer, and were able to show it could be made to work. A couple weeks later, I presented it to the rest of the development staff as their new unit testing practice.

Inevitably, they freaked out. As an outsider, there was no way that I could be right. I had taken too trivial of an example. I hadn’t considered X, Y and Z. You name it. But the presentation of a crude solution had an interesting effect. Someone on the team piped up and asked why the solution didn’t use the available tool. Someone else proposed changes. Soon I had a group of about 5-7 developers who all had ideas about how it could be done.

Two weeks later we had a redesigned, still imperfect, still somewhat crude (but less so that my original) solution. But this time, the team owned it and was willing to try it out. The key developers could get behind it and support it. It stood a chance of actually working.

While a quick and crude change may not be ideal, sometimes it can be exactly the thing to spur on more and better changes. If you never try to roll out a change at all, resistance is easy… simply talk it to death and convince yourselves that it cannot be done. But, if you try something, no matter how crude, and it works, then it serves to instantly destroy the perceived barriers and you finally have a foothold for future improvement.

We have seen the enemy, and he is us

I was just reading a great article, despite its age, on why software developers don’t practice what they preach.  One of the keys, as Humphrey points out, is that despite being taught best practices, we often view the programs as too small to deserve the attention of a full development lifecycle.  Furthermore, we (professors of college courses and management presumably) focus on outcomes rather than on the process to get there.

As he says in the article, “This craft-like approach for advancing the state of software practice has not been effective in the past and it will almost certainly not be effective in the future.”  And yet, we hear repeatedly of the software-as-a-craft viewpoint.

Moreover, he goes on, “As opposed to laboratory courses in chemistry, physics, or biology or the rigorous training of surgeons, software practices are largely ad-hoc and informal. Students thus become convinced that coding and testing are essential and everything else is optional. As they gain experience, it grows increasingly difficult to change their minds.”  We see this in current practices espoused – test driven development, pair programming, automated code review and unit test tools, continuous integration and testing… all focused on coding and testing and very little on designing a good solution in the first place.  Little attention is paid to boring, but proven practices such as Fagan-style inspections for analysis, design and coding.

While Humphrey’s research predates the current Agile movement, it is not reason to ignore the results of his research.  Process has a place, and can show measurable advantage over coding and testing alone, both in terms of quality and speed.

If we desire to have rigor in our processes, then it is up to us as management to demand it and audit for it.  We are in control of the outcomes if we choose to be.  To get there, we need to spend less time looking over the budget spreadsheets and more time looking at exactly how our developers get things done.