The allure of tools

Over the weekend I helped my brother install chair rail and wainscoting around his entire downstairs. I’ve done chair rail before, but I’d never done wainscoting. Still, I had a decent idea of how it was supposed to be done, so I was happy to help.

He has a relatively new house, so compared to my antique home, the walls would be straighter and easier to do the work on I figured. For the chair rail I simply set up the power mitre saw on a workbench in the garage and proceeded to make cuts as we needed them. We didn’t need to batch cut anything because we were mostly able to use full length pieces or cut sections to fit between windows and so on. It took about five hours to complete that stage, and we went to bed feeling pretty satisfied with our work.

The next morning we started on the wainscoting. For each section of wall we would be building picture frame-like boxes out of quarter round stock. I figured that this would be a great time to batch process parts. After all, new walls ought to lead to consistent heights for the boxes and I could simply cut thirty or forty vertical pieces quickly. So, first I set out to make a jig for the mitre saw. That required a run to the local hardware store for wood.

I’m still cautious so I measured the first vertical piece by hand and discovered it was no small task to cut on the saw. First you had to measure the length and cut the piece square. Then you had to rotate the saw to 45 degrees to trim one end. And then you had to rotate it 45 degrees in the other direction and trim the other end. Each time you rotate the mitre saw (which is older and not well oiled), it takes time and effort. The process of cutting even a couple vertical pieces was slow. Then, I’d have to walk up from the garage with the piece, check that it fit well and if not go back down and trim it again and come back up and down and up and down…

It took a really long time just to do the first three boxes. I finally said to my brother, go to the hardware store and buy a manual mitre box and saw. I could bring this little device into the room we were working on. Yes, each cut (the actual act of cutting a piece) took far longer, but consider all the waste I was able to remove from the process. I no longer had to walk up and down the stairs to the garage. I could simply measure the piece, walk a few steps to the mitre box, cut it and walk a few steps to check the fit. Setup of the mitre box is practically instantaneous. Simply pick the saw up and rotate it to one of the sets of slots for an angled or straight cut. No writing anything down. Since I don’t have to remember the length while I walk down to the garage, I no longer needed to write down the length of the cut. If I forgot by the time I got to the mitre box the cost of remeasuring was low.

By switching from a power mitre saw to a manual mitre box I saved a ton of time and effort. There’s a certain allure to power tools that isn’t always justified. The setup costs and limits on where you can put them can more than offset the advantage of one quick cut. It’s a great example of why you have to consider the whole process when designing a solution. Speeding up a tiny part of the process may incur costs that undo the benefit and then some.

Could a high standard deviation be good for your business?

If you had asked me a few weeks ago, I would have told you that having a high standard deviation in your process was never a good thing. After all, being unpredictable isn’t good. That is what is drilled into our heads all the time when you study Six Sigma.

The conversation started with a group of people discussing sports and, in particular, the Oakland A’s, who figure prominently into the book “Moneyball.” As stats geeks, we admire all that went into making the A’s a great team on a much smaller budget than many others. But the A’s are boring. You might get excited about Moneyball but, at least from where I stand, the A’s don’t fare well. If you use attendance as a measure, the A’s rank near the bottom – between 27th and 29th depending on the site you use.

So, how do we rationalize this disconnect. Solid data suggests that a far less monied team can compete, but the fans don’t show up. And the reality is, ultimately sports is a business just like any other and you need the fans to show up. So, why is it that a better average performance didn’t draw in the fans?

Well, because predictable is… Boring. There are places where predictable isn’t good. High standard deviation is good. What do people remember about the Red Sox? Eighty some odd years of coming close and failing, then the excitement of a few years of wins and we are back to the flame out stories. We like it when the game is exciting. And for winning to be exciting, there must be losses. Frankly, I’ll turn off a game if it is a blowout. If you could design a team who was reliably better than every other team, I bet nobody would watch. Predictable outcomes aren’t fun outcomes.

Don’t believe me? Check out the analysis of JCPenny’s “no more coupons” approach.. Lower prices are more predictable and overall should be better for customers, but it doesn’t work. Sometimes we have to appeal to something other than a reliable, repeatable experience. It is important to figure out when is the time to be predictable – a lot of the time, but not all the time.

How not to improve a process

One of the things that Six Sigma is all about is data driven decision making.  This seems like a good thing to do intuitively, but I think people skip it anyway because getting the data is difficult a lot of time.  Now, your experience probably won’t be as disastrous as the one I’m about to describe to illustrate my point, but it could get there…

Fortunately, my example has nothing to do with a real process I was working on.  I was in black belt training and we were using the Statapult to learn about various statistical techniques.  I am, by nature, super competitive so I wanted to win the competition that was set up between the various teams.  That meant putting the most balls on target and then presenting out your findings (your learning experience) to the teams at the end.

For those of you not familiar with the Statapult, I’ll provide some background.  Essentially, it’s a very small catapult that fires wiffle and pingpong balls.  It has a bunch of adjustments on it.  Here’s a picture of it:

A statapult

You have many adjustments you can make to it.  There’s the tension pin, the stop pin, the pullback angle, the rubber band connection point on the arm and the cup that holds the ball on the arm.  Two things need to be tackled with the statapult experiment.

  1. You have to figure out how the various adjustments affect the distance the ball flies.  After all, the goal is to launch the ball onto some target a random distance away.
  2. You have to figure out how to get the variation out of the system so that you can do #1.  This is critical.  There is a lot of noise in an uncontrolled statapult.  The rubber band is strong enough to move the entire statapult when released even if a couple hefty folks are holding it down.  And then there’s other things, like the tension pin which rotates in it’s slot as you pull back the rubber band.

In fact, it’s that very tension pin that taught me more about the statapult than anything else.  Here’s the important message that I want to demonstrate to you: do not improve a process based on your assumptions because you can do bad, bad things without realizing it.

After getting a day to play with the statapult, we decided (again, being very competitive) that we needed some serious improvements to control variation.  I went to the hardware store on the way home and bought:

  • 4 2 1/2 inch lag bolts
  • 16 washers
  • 4 wing nuts
  • A roll of foam insulating tape
  • 2 metal plates
  • A piece of 1/4″ plywood for a base (I had this in my garage)

When we came back in, I built a base to control our statapult’s movement.  We drilled holes in the plywood and pushed the bolts (with washers to prevent rip-out) through the board.  We then placed the statapult between the bolts and used the metal plates, more washers and wing nuts to hold the statapult to the base.  I attached the insulating foam to the underside of the metal plates to a) protect the statapult from damage (they’re insanely expensive) and b) to provide shock absorption. 

Finally we looked at what else we could do to take noise out of the system.  Well surely that spinning tension pin was a bad thing.  Every time we pulled back the arm it would rotate and the rubber band was clearly slipping over it.  This had to be a bad thing, so we clamped down the tension pin so it wouldn’t rotate.

By the time we were done with this, we were learning about Design of Experiments (DOEs).  We were doing a 2k DOE, which is pretty straightforward and we expected that when we were done that we’d have the best results out of the entire class.  After all, we had all the noise under control. 

The reality was that we had one of the worst results and we didn’t understand why.  The Master Black Belt teaching the class took a look at our data and concluded it was because of the firing angles we had chosen.  For a DOE to be effective it helps to have at least one continuous variable to adjust.  All the other adjustments on the statapult are discrete, so it is important to use a wide range of firing angles to get good fine control over the firing distance. Because of where we were in the classroom, if we used a wide range of options on the firing angle we’d launch the ball right into the wall.  This meant we couldn’t get a measurement on the shot and therefore wouldn’t be able to compute the results.  To compensate we used a firing angle of between 135 and 140 degrees for our experiments and it seemed like that might have not been enough variety in ranges.

Our MBB was kind enough to stay with us and do the experiment over with a wider range of firing angles (and a bit more room to fire) and see what we got.  The results still sucked.  It wasn’t the firing angle, but we discovered something very interesting.  When you do a DOE, you randomize the order in which you adjust all the parameters.  You do this to avoid introducing bias into the experiment.  You also take two measurements at each set of parameters.  So, if you were using settings A, B and C in the first shot, you might do 15 more shots with various settings before coming back around and doing settings A, B and C again.  Because it was getting late our MBB said to skip that randomizing because each time you do that you have to reset the whole device and it takes a while.  So, we were doing shot 1 and shot 2 at settings A, B, and C.  And we found out something crazy, the two shots weren’t producing anywhere near the same result each time.  If you do the same settings on the statapult and you’ve gotten rid of all the noise in the system it should produce the same result (or at least darn close to it).

That’s when we had an epiphany.  Locking down the tension pin rotation was a bad thing!  Yes, indeed the rubber band did move over the rotating pin but it was a good thing, not a bad thing.  By allowing the pin to rotate it equalized the tension on the rubber band on both sides of the pin.  Without it, sometimes the rubber band would be really tight and sometimes really loose and result in unpredictable shots.

Alas, it was now quite late and our MBB was unable to stay for round number 3.  One of my teammates and I did the entire experiment again with just the two of us (it’s much easier with 4 or more people).  And finally, we got a good DOE and a regression equation with a 95.6% R-Sqr (adjusted).

In the end, we didn’t win the competition.  It turns out there’s one more noise factor in the system we didn’t count on – having everyone else in class watch you during competition makes you really nervous.  We did come in second place, primarily because all our failures let us tell one of the best learning experiences about the statapult.

Getting back to my message for you of “do not improve a process based on your assumptions because you can do bad, bad things without realizing it” you can see how making bold assumptions about the statapult process resulted in us making the system work worse than it did if we just hadn’t played with it at all.  This is what you should take away about real-life process improvement.  It is not enough to think you know what’s right for the process, if you don’t understand how the process works, then don’t play with it.  Or do play with it, but have the fortitude to admit you screwed it up and now know enough to not do that again.